pandas-data-cleaning

# pandas Data Cleaning ## Overview Keywords: pandas, data cleaning, dtypes, missing values, NaN, imputation, duplicates, outliers, IQR, z-score, tidy data, melt, pivot, normalize, standardize, parse dates, categorical, data quality, ETL preprocessing. This skill turns a messy DataFrame into a tidy, correctly-typed, analysis-ready dataset using a repeatable, auditable workflow. The core principle: **profile first, decide explicitly, transform with logging, validate after.** Never mutate data silently — every fill, drop, or cast should be a deliberate, documented choice you can defend. Treat cleaning as a pipeline that produces (1) the cleaned DataFrame and (2) a record of decisions. Prefer chained, non-mutating transforms (`df.assign(...)`, `.pipe(...)`) over scattered in-place edits so the pipeline is reproducible top-to-bottom. ## Workflow 1. **Profile the raw data.** Before changing anything, understand it. Run `scripts/profile_data.py <path>` (or replicate inline) to get shape, dtypes, per-column null counts/percentages, unique counts, sample values, and candidate problems (mixed types, high-cardinality strings, numeric-looking objects, constant columns). See `references/cleaning-checklist.md`. 2. **Fix structure.** Standardize column names (snake_case, strip whitespace, dedupe). Set/verify the index. Drop fully-empty rows/columns and constant columns that carry no signal. Confirm one observation per row, one variable per column (tidy form). If not tidy, defer resha...