ai-data-curation-pretraining

# Pretraining Data Curation — Functional Reference Skill **Domain**: Building web-scale and synthetic pretraining corpora, running controlled data ablations. Distinct from applications-layer retrieval (RAG) and general data engineering. No theory. No generic pipeline intros. Focus on stage-by-stage decisions, heuristic thresholds, tooling choices, and ablation protocol. ## ASCII Flow ```text CommonCrawl WARCs | v [Extract] trafilatura / datatrove HTMLExtractor raw text + metadata (URL, timestamp, content-type) | v [Language ID] fastText lid.176.bin keep target language(s), threshold ≥ 0.65 | v [Quality Filter — Heuristic] Gopher / C4 rules symbol-to-word ratio, fraction lines ending ellipsis, stopword density, word count bounds, mean word length | v [Quality Filter — Classifier] FineWeb-Edu edu-score / custom trained on human labels; outperforms heuristics on recall | v [Near-Dedup] MinHash + LSH banding (datasketch) n-gram shingles -> MinHash signature -> band partitioning | v [Exact-Substring Dedup] suffix-array substring match remove exact repeated sequences across documents | v [Decontamination] n-gram match against eval benchmarks FAIL LOUD — contaminated eval numbers are the field's #1 silent failure | v [PII / Safety Scrub] regex + classifier email, phone, SSN, credit card patterns; hate/CSAM removal | v [Tokenize + Shard] HF tokenizers / tiktoken; Parquet shards | v [Domain Mix + Weight] dolma too...