data-modeling

# Data modeling ## Layers, and why the middle one matters Three layers, each with one job: 1. **Raw** — source data, append-only, otherwise unmodified. Do not apply *business* logic on ingest: you cannot recover what you discarded, and the logic will need to change retroactively. **Privacy and security transformations are the exception, and belong at ingest.** Credentials and secrets should never land in the warehouse at all. Personal data that is not needed should be dropped rather than stored and governed later, and identifiers you must keep but rarely need in the clear should be tokenized or encrypted on arrival. Retention and deletion apply from ingest, not from the marts. The distinction: strip what you must not hold, keep everything you are entitled to hold, and leave interpretation for later. 2. **Staging** — cleaned and conformed: consistent types, standardized names, deduplicated, no business logic yet. 3. **Marts** — business-facing models shaped for how questions are asked. The discipline that pays is keeping business logic out of layers 1 and 2. Logic embedded in ingestion cannot be changed retroactively, and it will need to change. ## Grain is the decision everything follows from State the grain of every table in one sentence: *one row per what*. "One row per order line per day" is a grain. "Order data" is not. Most modeling errors are grain errors, and they surface as fan-out — a join multiplying rows so every downstream sum i...