profilelisted

# Profile Phase 2 of FAW. **No design until there are numbers.** In software, requirements come from people. In data work, half of them are in the source and are only discovered by measuring. Every question answered by assumption produces an artifact that runs and returns the wrong thing. ## The rule that governs this phase **Every number goes with the query that produced it.** A number without a query does not enter the receipt. Profiling is read-only. If something has to be written, ask for explicit authorization first. ## What to measure ### A new source ```python # Rows df.count() # Candidate natural key. This is what decides whether it is a key total = df.count() unique = df.select(*KEY).distinct().count() print(f"rows={total} unique={unique} duplicates={total - unique}") # For every column the curated layer will consume df.select([ F.count(F.when(F.col(c).isNull(), c)).alias(f"{c}_nulls") for c in COLS ]).show() # Sentinel values in the source, with their real frequency for c in DATE_COLS: df.select(F.round(100 * F.avg((F.col(c) == SENTINEL_LIT).cast("int")), 2)).show() # Real types, not the ones in the specification df.printSchema() ``` **The natural key is proven, not copied from a specification.** What documents a source and what the source is diverge over time. A specification can name a column the table no longer has, or never had. The key is confirmed by querying the table. ### An existing curated layer (`MODEL` and `REPORT` tiers) - Rows