debugging-data-pipelineslisted

# Debugging Data Pipelines ## When to use - A pipeline job failed, or output data is wrong/missing/duplicated. - A dashboard is stale or a metric doesn't reconcile with the source. - A stakeholder reports a discrepancy and you must find the cause. - Do NOT use for tool-specific run errors already covered by `debugging-dbt-runs` / `debugging-airflow-pipelines` (start there, then use this for data-correctness incidents). ## Workflow ``` - [ ] Define the symptom precisely (which table, column, rows, time window) - [ ] Trace lineage upstream to find the first stage where data is wrong - [ ] Isolate: is it a code bug, bad input, late data, or a run failure? - [ ] Reconcile the suspect stage against its source (counts/sums) - [ ] Fix root cause, then plan an idempotent backfill of affected windows - [ ] Add a check so it can't recur silently ``` 1. **Pin the symptom.** "Revenue for 2026-01-15 is ~30% low in `fct_orders`" is debuggable; "numbers look off" is not. Get the table, column, rows, and window. 2. **Trace lineage upstream.** Walk from the wrong output backward through models/ tasks to find the **first** stage where the data is already wrong. Binary-search the DAG rather than reading every stage. 3. **Classify the cause:** code change, bad/late source data, a failed/partial run, or a non-idempotent duplicate. 4. **Reconcile** the suspect stage vs its input (row counts, key counts, sums) to confirm where the delta appears. 5. **Fix + backfill** the aff