a-priced-bias-is-a-work-item-not-a-caveatlisted

Use at implementation, experimentation and analysis on a task graded by an error metric over a predictions file, when a diagnostic you ran after freezing your design says the numbers you are about to ship are biased - too high, too low, on the wrong scale, in the wrong units - and a pre-registered decision rule is the reason you are recording it rather than fixing it. Covers the one test that separates a forbidden search over candidates from an ordinary bias correction, where the correction may be estimated, the held-out check that has to pass before you apply it, and how to ship the corrected file while still reporting the frozen verdict.
tangxiangru/AutoR · ★ 805 · AI & Automation · score 79

Install: claude install-skill tangxiangru/AutoR

# A number that prices a defect in the deliverable is a work item Freezing a decision rule before the results exist is the right habit and it protects exactly one thing: the mapping from a statistic to a verdict. It stops you choosing, after the fact, the comparison that happened to come out well. The graded file is not a verdict. It is an artifact, the grader has no notion of your protocol, and no statistic is being adjudicated by the numbers inside it. So when a diagnostic run after the freeze says those numbers are biased, the freeze has an opinion about what you may *conclude* and no opinion at all about what you may *ship*. Runs conflate the two, and the conflation is expensive: the discipline written to stop you fooling yourself becomes the reason you knowingly ship a defect you have already measured. ## 1. The test, in one question > **Does the correction require me to choose among candidates using their scores?** - **Yes** — that is a search, the freeze forbids it, and it is right to. Swapping the shipped model for the one that came out best in a comparison you ran after the freeze is selection, whatever you call it. - **No** — it is a bias correction, and the freeze has nothing to say. Two operational forms of the same question, either of which settles it in a minute: 1. **Would I apply exactly this correction whichever candidate were in the file?** If yes, it is a property of the data or of the estimator class, not of a leaderboard. 2. **Does applyi