ml-experiment-standards

# ML Experiment Standards Purpose: every ML job (quick prototypes included) is reproducible, leakage-free, and metric-justified. These are not optional polish; every skipped item typically returns as "the model collapsed in production" or "the result didn't replicate". ## 1. EDA comes first Before any model, produce and show: distributions, missingness rates, outliers, target balance, salient correlations. Metric and loss choice depend on this information; a model recommendation without EDA is a guess. ## 2. Leakage audit At every split decision, answer explicitly (and write the answer as a code comment): "Does the training set contain indirect information about any test sample?" | Data type | Correct split | Why | |---|---|---| | Independent samples | Stratified k-fold | Preserves class ratios | | Time series | TimeSeriesSplit / walk-forward | Future must not leak into past | | **Spatial data** | Spatial block CV — see `references/spatial-cv-protocol.md` | Neighbors are near-duplicates | | Grouped data (patient, parcel, scene) | GroupKFold | A group must not straddle the split | - Scalers/encoders/imputers are **fit on train only**; the clean path is `sklearn.pipeline.Pipeline` — CV then fits correctly by construction. - Target-derived features (target encoding etc.) must be computed out-of-fold, and shown to be. The spatial protocol in `references/spatial-cv-protocol.md` is the single canonical source for this repo — other skills link here; do not restate it. #...