an-oracle-ceiling-is-not-headroom-yetlisted

# An oracle ceiling is not headroom until the input predicts the oracle's choice The most persuasive number a run produces about itself is a ceiling: hold everything fixed, let one degree of freedom take its best possible value on every row, re-score. "+0.0276 available from getting the output length right per example" is concrete, measured, and reads in a plan like a plan. It is also computed with the label in hand, and that is the whole problem. A ceiling says how far the metric *would* move. It says nothing about whether anything you are allowed to see moves with it. Turning one into a decision takes four cheap steps, and the two in the middle are the ones that get skipped. ## 1. The ceiling Replace the lever with its per-row optimum, everything else fixed, and score. On a subsample this is minutes. Do it per lever **and** for the levers jointly, because they overlap: in one measured case two levers were worth +0.0276 and +0.0118 alone and +0.0419 together — which is close to saying the second is the first one wearing a different name, and a plan that budgets an hour for each has double-counted. ## 2. Invert the ceiling into the correlation you would need You will never reach the ceiling; you will reach some fraction of it set by how well you can predict the oracle's per-row choice. So simulate that directly. Build synthetic predictors of the oracle's choice at a ladder of target correlations ρ = 0.1, 0.2, … 0.9 — take the oracle's own choice and add noise scaled to