eval-set-builderlisted

# Building an eval set that can actually answer the question Most eval sets are assembled to a round number. Someone picks 50 or 100 items because those feel like enough. Then a 2% error budget gets claimed off a sample that could never have detected a 2% error rate in the first place. ## The refusal **If the sample is too small to certify the budget, say so and stop.** Do not report a pass. "No errors found in 60 items" is a fact about the sample, not about the model. Rule of three: with zero observed errors in n items, the 95% upper bound on the true error rate is roughly 3/n. Certifying "under 2%" therefore needs about 150 clean items. Under 1% needs 300. ```bash node lib/stats.ts # see certifies() and minItemsToCertify() ``` Run `certifies(errors, n, budget)` and report `upperBound`, not the point estimate. ## Procedure 1. **Get the budget first.** Ask what error rate the process can absorb before anyone picks a sample size. If nobody can answer, that is the finding. Stop there. 2. **Compute the floor.** `minItemsToCertify(budget)`. State it before any data is collected, so the number is not negotiated after the fact. 3. **Sample from real traffic, not from imagination.** Generated or hand-written items are easier than production and will overstate accuracy. If only synthetic data is available, label the whole result as non-evidential. 4. **Stratify by the slices that matter** (channel, language, document type, region). An aggregate that hides a c