a-priced-bias-is-a-work-item-not-a-caveat

# A number that prices a defect in the deliverable is a work item Freezing a decision rule before the results exist is the right habit and it protects exactly one thing: the mapping from a statistic to a verdict. It stops you choosing, after the fact, the comparison that happened to come out well. The graded file is not a verdict. It is an artifact, the grader has no notion of your protocol, and no statistic is being adjudicated by the numbers inside it. So when a diagnostic run after the freeze says those numbers are biased, the freeze has an opinion about what you may *conclude* and no opinion at all about what you may *ship*. Runs conflate the two, and the conflation is expensive: the discipline written to stop you fooling yourself becomes the reason you knowingly ship a defect you have already measured. ## 1. The test, in one question > **Does the correction require me to choose among candidates using their scores?** - **Yes** — that is a search, the freeze forbids it, and it is right to. Swapping the shipped model for the one that came out best in a comparison you ran after the freeze is selection, whatever you call it. - **No** — it is a bias correction, and the freeze has nothing to say. Two operational forms of the same question, either of which settles it in a minute: 1. **Would I apply exactly this correction whichever candidate were in the file?** If yes, it is a property of the data or of the estimator class, not of a leaderboard. 2. **Does applyi...