add-the-baseline-back-on-the-split-you-cannot-score

Solid

Use at study design, implementation and every write thereafter, whenever the value you submit is assembled from parts - a fitted baseline plus a model's residual, a level plus a shape, a de-trended prediction that has to be re-trended, any inverse transform - and the validation arrays and the graded arrays are produced by separate calls. Covers assembling every split through one function, using the baseline you already fitted as a label-free reference vector on the graded split, why row count, header, dtype and finiteness cannot see this class of error, and putting the gate inside the writer rather than in a script somebody has to remember to run.

AI & Automation 805 stars 25 forks Updated 2 weeks ago NOASSERTION

Install

View on GitHub

Quality Score: 82/100

Stars 20%
97
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Add the baseline back on the split you cannot score When the prediction is `base + residual`, it gets assembled twice: once on your validation split, where you immediately compare it to labels, and once on the graded split, where you compare it to nothing. Only the first is checked. Every diagnostic runs on the first. Every selection decision is made on the first. The file that is scored comes from the second. Drop the `base` term on the graded path and not the validation path and you get a submission that passes every check a run normally has - right row count, right header, all finite, correct dtype - behind a validation number that is not merely plausible but *better than the incumbent*, because that number was computed from the correctly assembled validation array. The file is wrong by a multiplicative constant. Under an error metric with no upper bound that is not a regression; it is near the worst score the task can produce. This is the specific failure that the standard advice cannot reach. A brief that warns you a malformed file scores nothing, and a helper that enforces shape and finiteness, are both correct and both silent here. `select-on-the-file-you-would-ship` says the selection number must come out of the writer's code path; that is necessary and not sufficient, because it is satisfiable on the split you can score while the split you ship is assembled by a different call. ## Rule 1 - one assembler, mapped over the splits Do not write two branches. Write ...

Details

Author
tangxiangru
Repository
tangxiangru/AutoR
Created
6 months ago
Last Updated
2 weeks ago
Language
Python
License
NOASSERTION

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

a-scoreable-file-in-the-first-hour

Use at the first stage of a run whose deliverable is a predictions file, and again at every stage when one still does not exist. Covers why a trivial submission written early dominates a good one written late, what the first version should contain, and how to improve it in place without ever leaving it invalid.

805 Updated 2 weeks ago
tangxiangru
AI & Automation Solid

chemistry-reproduce-the-scoring-path-before-you-replace-it

Use at implementation, experimentation and analysis when you are reproducing a published benchmark number and the source's scoring path is one you can read — which rows are scored, in what order, how many the loader drops, which epoch is reported, how tasks are pooled, over how many seeds. Covers implementing that path exactly before improving it, the one-row-per-step ladder from the published rule down to your own honest estimate, and why one un-replicated step makes the reproduction gap you report uninterpretable.

805 Updated 2 weeks ago
tangxiangru
AI & Automation Solid

calibrate-the-level-on-the-window-you-cannot-score

Use at implementation and afterwards whenever the rows you will be scored on lie outside every window you can check against truth — a forecast horizon that starts where the supplied history ends, a later time period, a different site, batch or cohort, a test split whose label column has been removed — and your only bias check was run on a backtest fold or a random validation split. Covers why "my predictions are unbiased" is a statement about the folds and not about the graded rows, how to measure the overall level of your predictions on the graded rows with no labels at all, why the level ratio is only the alarm and a metric scan is the number, what to do when the two windows disagree, and when a low forecast is correct rather than broken.

805 Updated 2 weeks ago
tangxiangru