data-engineering-bigquery-gcslisted

# Data Engineering: BigQuery & GCS Pipelines Stack-agnostic BigQuery data pipeline patterns with medallion layering and Temporal orchestration. ## When to use - Implementing medallion architecture (bronze/silver/gold layers) for a data warehouse - Building ETL pipelines that load data to BigQuery - Designing BigQuery schemas with date partitioning and clustering - Orchestrating data sync jobs with Temporal workflows and activities - Implementing atomic upserts (MERGE) from staging tables to production tables - Loading in-memory pandas DataFrames directly to BigQuery (bypassing GCS staging) - Implementing CDC deduplication for Datastream-synced bronze tables - Evolving table schemas dynamically by adding nullable columns - Executing parameterized BigQuery queries safely without SQL injection - Streaming small batches of rows into BigQuery tables - Generating table and column descriptions for data catalogs ## Core conventions ### Medallion Architecture (Bronze → Silver → Gold) **Three-layer pattern**: Bronze = raw ingestion (Datastream CDC, GCS exports); Silver = cleaned, joined, business logic; Gold = aggregated metrics and KPIs. Each layer is a separate BigQuery dataset. **Bronze table conventions**: Schema `bronze`, table prefix `br_`, tags `["bronze", "cdc"]`. Use `QUALIFY ROW_NUMBER() OVER (PARTITION BY {pk} ORDER BY created_at DESC, datastream_metadata.source_timestamp DESC) = 1` to deduplicate CDC events. Exclude deletes: `WHERE datastream_metadata.change_type NOT