dsql-to-analytics-pipelinelisted

Build a real-time analytics pipeline downstream of Aurora DSQL using the DSQL CDC stream. Covers consumer Lambda correctness, the append-only event log + ROW_NUMBER reconstruction pattern that absorbs unordered/duplicate delivery, Redshift Data API parameter-cap chunking, async statement polling, BisectBatchOnFunctionError, MaximumRecordAgeInSeconds, DLQ wiring, and the DSQL-preview gotcha that both INSERT and UPDATE arrive as op='c'. Triggers on phrases like: DSQL to analytics, DSQL to Redshift, analyze DSQL data, real-time DSQL analytics, DSQL CDC pipeline, Aurora DSQL change data capture, stream DSQL changes, replicate DSQL, DSQL to S3, DSQL to Iceberg, DSQL Kinesis Lambda, DSQL data warehouse.
jaingxyz/aws-data-skills · ★ 0 · Data & Documents · score 72

Install: claude install-skill jaingxyz/aws-data-skills

# Aurora DSQL to Analytics Pipeline Skill This skill covers everything DOWNSTREAM of the DSQL CDC stream: how to wire the stream into Kinesis, write a correct consumer Lambda, model events at the analytics sink, and avoid the data-loss footguns that come with async statement APIs and unordered delivery. Source-side DSQL concerns (DDL, IAM auth, OCC, schema design) belong to the companion `dsql` skill. ## Why this skill exists In the Aurora DSQL CDC public preview, both INSERT and UPDATE arrive as `op='c'`. Code that expects a separate `'u'` op (the typical Debezium / DMS shape) will silently treat every UPDATE as if no row existed, then miss the latest state when downstream readers query a naive `MAX(event_id)` view. The append-only + ROW_NUMBER-by-commit-timestamp pattern this skill prescribes handles both cases correctly. There are several sibling footguns (the Redshift Data API's 200-parameter cap, the async `execute_statement` poll requirement, poison-record wedging) that cause silent data loss when missed. Each one is called out below. ## When to use - You have an Aurora DSQL cluster and want the change events available for analytics in Redshift Serverless, S3 Tables (Iceberg), or both. - You are writing the consumer Lambda that reads from the DSQL CDC Kinesis stream and lands rows in an analytics sink. - You are choosing between an in-place upsert and an append-only event log for the sink table shape. - You need the operational gotchas (batch sizes, retries,