cost-the-rung-you-need-not-the-cheapest-one-in-the-family

# Cost the rung you need, not the cheapest one in the family Your survey produced a ladder: the published numbers for this dataset and this metric, each with the kind of model that produced it. The top of the ladder is what you are graded against. You cannot obviously afford it, so you build something, time an epoch, and refuse. That refusal is the most expensive artifact this shape of run produces, and it is almost never wrong about the clock. It is wrong about **what got timed** and **how much of it you actually needed**. ## Error one: you priced a rung that could not have helped you Before timing anything, put the ladder next to the number you already hold. Every rung whose published value is worse than your current validation score is out of the decision: even free, it cannot improve the deliverable. Timing one answers no question you have. And a price is a fact about the model you ran. It does not travel up the ladder. Rungs of one family differ by what the representation resolves -- distances, then directions, then oriented vectors -- and by depth and width, and each of those moves cost by a large factor in either direction. A cheap-rung timing extrapolated into a rejection of an expensive rung is an estimate with no measurement under it, wearing a measurement's clothes. > Measured. A run holding **11.977** on the metric built a cut-down > message-passing network -- 135k parameters, 3 layers, hidden 96 -- timed it at > 107.9 s/epoch, and registered the hypothesis ...