a-cut-point-is-a-fitted-parameter-not-a-setting

# The cut point is a model parameter. Fit it like one or you guessed it Most submissions of this shape contain a scalar somebody typed: the margin above which you commit an answer, the probability above which you flag a row, the confidence below which you fall back to a default. It looks like configuration. It is a parameter of the shipped model, and it frequently moves the score more than the model does, because it sets the label on every row in the tail — which is where all the undecided rows are. Three things go wrong with it. They are independent and they compound. ## 1. It is fitted on the smallest sample you own and applied to the largest You fit the model on tens of thousands of rows, then pick the cut point off whatever labelled subset you happened to score — a few hundred, chosen because that is what inference finished on. That estimate is worse than its sample size suggests. You are taking an **argmax over a grid**, not a mean: the noise is selected for, so the peak of an accuracy-versus-cut-point curve on 250 rows is both biased upward in sample and badly located out of it. What to do: - **Write the row count next to the number.** `thresh=-11.5 (fitted on n=250)`. If that count is two orders of magnitude smaller than the split you apply it to, this is the least-evidenced thing in your submission and you now know it. - **Refit whenever more labelled rows exist.** Refitting costs no inference — the scores are already on disk — so it is the cheapest scor...