firehose-iceberg-pipelinelisted

Build an Amazon Data Firehose delivery stream that writes to Apache Iceberg tables on S3 Tables via IcebergDestinationConfiguration. Covers the column-shape footgun (records land in the error bucket with Iceberg.MissingColumnWithinRecord unless a transform Lambda reshapes them), the microsecond timestamp requirement, the three-phase deploy pattern that works around Firehose's synchronous glue:GetTable validation at create time, parameter-inheritance traps in `aws cloudformation deploy`, Lake Formation grant ordering, and how to decode the error bucket. Use when you see "Firehose to Iceberg", "Firehose to S3 Tables", "IcebergDestinationConfiguration", "Firehose data transformation Lambda", "Firehose error bucket", "Iceberg.MissingColumnWithinRecord", "MissingColumnWithinRecord", or any time someone is wiring a Kinesis Data Stream / Direct PUT source to Iceberg tables and producer record shape does not match the Iceberg column layout.
jaingxyz/aws-data-skills · ★ 0 · DevOps & Infrastructure · score 69

Install: claude install-skill jaingxyz/aws-data-skills

# firehose-iceberg-pipeline Specialty skill for one architecture choice: **Amazon Data Firehose with `IcebergDestinationConfiguration` writing to Apache Iceberg tables on S3 Tables**, with optional reshape via a transform Lambda. ## When to use Load this skill when: - You are setting up `AWS::KinesisFirehose::DeliveryStream` with `IcebergDestinationConfiguration` (CFN), or the equivalent `IcebergDestinationConfiguration` block in the `firehose:CreateDeliveryStream` API. - The Firehose source is either a Kinesis Data Stream (`KinesisStreamAsSource`) or `DirectPut`. - The destination is an Iceberg table managed by Amazon S3 Tables (`AWS::S3Tables::TableBucket` + namespace + table), exposed via the bucket-nested `s3tablescatalog/<bucket>` Glue catalog. - You see `Iceberg.MissingColumnWithinRecord` in the error bucket and every record is failing. - You hit `Role ... is not authorized to perform: glue:GetTable for the given table or the table does not exist` during stack create. ## When NOT to use - The Iceberg table lives in a regular S3 bucket with the standard Glue catalog (not S3 Tables). The IAM, ARN, and Lake Formation patterns here assume the bucket-nested `s3tablescatalog` federation; a generic Iceberg-in-S3 setup uses the default catalog and different grants. - You are reading the Iceberg table from Amazon Redshift Serverless via `CREATE EXTERNAL SCHEMA`. That belongs in `lakehouse-redshift`. This skill stops at "Firehose is delivering rows