← ClaudeAtlas

pipeline-verifylisted

Verify a data pipeline in a dev environment — run it against a bounded sample and assert schema, counts, idempotency, and clean logs. The data-platform counterpart of driving a web app in a browser.
SilviaAre95/wayworks · ★ 1 · Data & Documents · score 77
Install: claude install-skill SilviaAre95/wayworks
# Pipeline Verify Verify in dev: **$ARGUMENTS** Unit tests prove functions; this proves the pipeline. Run the real thing end-to-end against a bounded sample and observe what comes out. ## Steps 1. **Bound the input**: pick a sample slice — a fixed date partition, N rows, or a fixture dataset. State exactly what goes in (source, row count, date range) before running. Never run unbounded. 2. **Run the pipeline** end-to-end in the dev environment (local runner, dev project, staging warehouse). Capture logs and runtime. 3. **Assert the output**: - **Schema**: output matches the destination contract — column names, types, nullability. Flag any drift. - **Row accounting**: rows in vs rows out vs rows rejected must reconcile. `in != out + rejected` means silent loss — that is a failure, not a footnote. - **Quality**: null rates and duplicate rates on key columns within expected bounds; spot-check 3–5 records field-by-field against the source. - **DLQ**: empty, or every record in it explained. 4. **Prove idempotency**: run the pipeline a second time on the same input. Destination state must be unchanged (same counts, no duplicates). A pipeline that double-loads on re-run fails verification. 5. **Check the logs**: no unexplained errors or warnings; freshness/monitoring hooks fired if configured. ## Output Format ```markdown ## Pipeline verification: <name> **Input**: <sample description, N rows, partition> **Runtime**: <duration> | Check | Result | Detail | |--