building-kafka-consumerslisted

Build reliable Apache Kafka consumers and producers — consumer groups and partition assignment, offset commit strategy, at-least-once vs exactly-once, idempotent/transactional producers, rebalancing, and dead-letter handling. Use when writing Kafka consumers/producers, configuring offset commits or consumer groups, tuning throughput, or handling rebalances and poison messages.
Unknown-333/awesome-data-engineering-skills · ★ 16 · Data & Documents · score 68

Install: claude install-skill Unknown-333/awesome-data-engineering-skills

# Building Kafka Consumers ## When to use - Writing or debugging Kafka consumers/producers. - Choosing offset-commit strategy and delivery guarantees. - Tuning consumer-group parallelism, rebalancing, or dead-letter handling. - Do NOT use for stream processing/windowing (use `processing-streaming-data`). ## Workflow ``` - [ ] Size partitions to target parallelism (consumers <= partitions) - [ ] Commit offsets AFTER successful processing - [ ] Make the sink idempotent (upsert by event key) - [ ] Handle rebalances (commit on revoke, avoid long poll gaps) - [ ] Route poison messages to a dead-letter topic ``` 1. **Partitions cap parallelism.** A consumer group scales out only up to the partition count; extra consumers sit idle. Choose partitions for peak throughput. 2. **Commit after processing.** Commit offsets once the work is durably done, not before — committing early loses messages on a crash. 3. **Idempotent sink.** At-least-once means duplicates on retry; upsert by a stable event key so reprocessing is harmless. 4. **Rebalances happen.** Commit on partition revoke and keep `poll()` intervals under `max.poll.interval.ms` so the broker doesn't evict the consumer. 5. **Poison messages** go to a dead-letter topic with the error, so one bad record doesn't block the partition. ## Patterns **Manual commit after processing:** ```python consumer = KafkaConsumer("orders", group_id="etl", enable_auto_commit=False,