cdc-streaming-pipelinelisted

Build a real-time change-data-capture pipeline from a transactional database (Aurora DSQL, Aurora Postgres, RDS) to Redshift Serverless, S3, or any other sink, via Kinesis Data Streams (or MSK). Covers producer setup, consumer Lambda correctness, the append-only event log + reconstruction pattern, schema-drift handling with SUPER + JSON_PARSE, and the operational gotchas (Redshift Data API parameter caps, async statement polling, poison-record handling, retry-and-bisect). Use when the user asks for "CDC to Redshift", "stream change data capture", "Aurora CDC", "DSQL CDC", "Postgres replication to S3", or "build a Lambda consumer for Kinesis CDC". Aurora DSQL public-preview specifics are flagged; the rest is source-agnostic.
jaingxyz/aws-data-skills · ★ 0 · DevOps & Infrastructure · score 69

Install: claude install-skill jaingxyz/aws-data-skills

# CDC streaming pipeline (Kinesis -> Redshift Serverless / S3) This skill captures the lived patterns for building a real-time CDC pipeline: a transactional source publishes change events to Kinesis, a Lambda consumer applies them to a sink (Redshift Serverless event log, or S3 / Iceberg via Firehose). It is source-agnostic: the producer-side notes are split into "any CDC source" and "Aurora DSQL preview" callouts. ## When to use - You are wiring up a CDC pipeline from Aurora DSQL, Aurora Postgres, or RDS to Kinesis Data Streams (or MSK) with a Lambda consumer. - You need to write the consumer Lambda that pushes CDC rows into Redshift Serverless via the Redshift Data API. - You are deciding how to model CDC events at the sink (append-only vs in-place upsert, how to absorb schema drift). - You want the operational gotchas (parameter limits, async statement polling, retries, poison records) before you debug them. ## When NOT to use - You only need the lakehouse / cold-path target (S3 Tables, Iceberg, Redshift external schema, federated Glue catalogs). Use the `firehose-iceberg-pipeline` and `lakehouse-redshift` skills for those. - You want generic Redshift Serverless setup unrelated to CDC (workgroup sizing, RPU pricing, snapshot policy). Out of scope here. - You are doing batch ETL / DMS full-load. CDC is for ongoing change capture; bulk loads belong elsewhere. - You need EMR / Glue / Spark transforms inside the stream. This skill uses Lambda + Redshift