engineering-databricks-pipelineslisted

# Engineering Databricks Pipelines ## When to use - Building Databricks jobs, notebooks, or Delta Live Tables (DLT) pipelines. - Working with Delta Lake (MERGE, OPTIMIZE, time travel, schema evolution). - Ingesting files incrementally with Auto Loader. - Organizing data under Unity Catalog (catalog.schema.table) and sizing clusters. - Do NOT use for generic Spark tuning (use `optimizing-pyspark-jobs`). ## Workflow ``` - [ ] Model tables as Delta under Unity Catalog (catalog.schema.table) - [ ] Ingest raw with Auto Loader (incremental, schema-tracked) - [ ] Transform in medallion layers (bronze -> silver -> gold) - [ ] Use MERGE for idempotent upserts; OPTIMIZE/Z-ORDER for read speed - [ ] Right-size the cluster/job; enable Photon for SQL-heavy work ``` 1. **Delta + Unity Catalog** are the defaults: ACID tables with governance, lineage, and access control. Use three-level names `catalog.schema.table`. 2. **Auto Loader** (`cloudFiles`) ingests new files incrementally and tracks schema, avoiding full re-lists of cloud storage. 3. **Medallion layers** — bronze (raw), silver (cleaned/conformed), gold (aggregated marts) — keep transformations testable and replayable. 4. **MERGE** makes loads idempotent; **OPTIMIZE** + **Z-ORDER** on filter columns speed reads. ## Patterns **Idempotent upsert with Delta MERGE:** ```python from delta.tables import DeltaTable (DeltaTable.forName(spark, "main.sales.fct_orders").alias("t") .merge(updates.alias("s"), "t.order_id =