implementing-cdc-with-debeziumlisted

# Implementing CDC with Debezium ## When to use - Replicating an operational database (Postgres/MySQL/SQL Server) to a warehouse/lake in near real time. - You need **deletes** and every intermediate change (watermark extraction can't see deletes). - Consuming or applying a Debezium change stream idempotently. - Do NOT use for simple periodic batch pulls (use `building-ingestion-pipelines`). ## Workflow ``` - [ ] Enable the DB log (Postgres logical replication / MySQL binlog / MSSQL CDC) - [ ] Configure the Debezium connector (tables, snapshot mode, keys) - [ ] Handle the initial snapshot, then streaming changes - [ ] Apply changes idempotently: MERGE keyed on PK, ordered by log position - [ ] Handle deletes (tombstones) and schema changes ``` 1. **Enable the log.** Debezium reads the DB transaction log: Postgres logical replication (`wal_level=logical` + a publication/slot), MySQL binlog (`ROW` format), or SQL Server CDC. Grant the connector the needed privileges. 2. **Configure the connector** with the tables to capture, the snapshot mode, and the primary key. It emits an initial **snapshot**, then live **change events**. 3. **Apply idempotently.** Each event carries `before`/`after`/`op` and a log position (LSN/GTID). MERGE on the primary key and order by the position so out-of-order or replayed events converge to the correct state. 4. **Deletes** arrive as `op=d` (plus a null-value tombstone for log compaction); apply as a delete or soft-delete f