implementing-data-quality-checkslisted

# Implementing Data Quality Checks ## When to use - Adding validation so bad data is caught before consumers see it. - Setting up freshness, volume, schema-drift, or integrity checks. - Choosing a tool (dbt tests, Great Expectations, Soda) and where to place checks. - Do NOT use for dbt-specific test syntax only (use `testing-dbt-projects` for the dbt details). ## The six dimensions to cover 1. **Freshness** — did the data arrive on time? (max load timestamp vs SLA) 2. **Volume** — is the row count in the expected range? (catches partial loads and duplication) 3. **Schema** — did columns/types change unexpectedly? (drift) 4. **Completeness** — required fields not null. 5. **Uniqueness / integrity** — keys unique, foreign keys valid. 6. **Validity / distribution** — values in allowed ranges/sets; no sudden distribution shifts. ## Workflow ``` - [ ] Put blocking checks at the entry point (raw/staging) so bad data stops early - [ ] Cover the six dimensions on critical tables - [ ] Decide warn vs block per check by business impact - [ ] Route failures to alerts + a quarantine/failure store - [ ] Add reconciliation between source and target for critical flows ``` 1. **Check early.** Validate at ingestion/staging so bad data fails fast rather than propagating into marts and dashboards. 2. **Cover the six dimensions** on tables that feed decisions/SLAs. 3. **Warn vs block.** Block (fail the run) on integrity-critical checks; warn on soft signals. Blocking everyt