data-engineering-disciplinelisted

Discipline guardrails for data-engineering work with downstream consumers — activate at the START of the task, before writing code, because silent semantic drift is the dominant risk. Activate on: migrating or porting a pipeline, refactoring a transform, backfilling or replaying history, evolving a schema (add / rename / retype / drop a column), creating a new dataset — or a metadata / catalog / lineage emitter whose output a separate tool loads — that has consumers, designing or reviewing a data contract, reshaping a tool / API response payload a client depends on, writing or changing the tests, fixtures, or expected values that gate a pipeline, generating pipeline code with an LLM, and investigating a consumed dataset that misbehaves — "the numbers changed / look different", or a table/extract that "ran but didn't update / is stale / isn't refreshing / the watermark didn't advance". A hand-authored schema-as-data document counts when code is generated from or validated against it; not when its only consumer
grimaldost/craft-collection · ★ 2 · Data & Documents · score 68

Install: claude install-skill grimaldost/craft-collection

# Data Engineering Discipline The output of a data pipeline is a contract. Columns, dtypes, row cardinality, null behavior, group cardinality, and semantics are all part of that contract, whether anyone wrote it down or not. This skill defends against **silent breakage** — output that compiles, runs, and looks plausible, but no longer means what consumers think it means. Every principle in this skill exists because one of those guarantees has historically been broken in a way that cost real money or real trust. This skill is scenario-agnostic. The same discipline applies to migrating a pipeline, creating a new one, refactoring an existing one, evolving a schema, backfilling history, designing incremental loads, and investigating downstream regressions. The specifics change; the non-negotiables do not. ## Scoped-change lane A bounded change to one transform or seam — a single tz-cursor fix, one emitter field, a localized incremental load — does not need the full-migration apparatus. Pin the contract for *that* seam (its schema / dtypes / semantics and the consumers of the changed field), run only the parity and real-data checks that touch it, and leave the rest of the pipeline alone — a wide scope is what invites the "improving while executing" failure mode. The four non-negotiables below still hold for the seam; what shrinks is the blast radius you verify, not the rigor. Follow the project's own conventions over greenfield defaults (python-engineering's edit lane owns tha