public-kafka-expert-baselisted

# Kafka Knowledge Base Foundational guidance for building event streaming applications with Apache Kafka. > **Source**: Adapted from [confluentinc/agent-skills](https://github.com/confluentinc/agent-skills) and Apache Kafka best practices. ## Topics & Partitions - Design topics around domain events (`order.created`, `payment.completed`), not database tables. - Partition count determines max consumer parallelism — start with 3-6, scale up as needed. - Partitions are ordered; messages within a partition have guaranteed order. - Use partition keys for related messages that must be ordered (e.g., `user_id` for user events). ## Consumer Groups - All consumers in a group share the workload (each partition → one consumer). - Multiple groups can independently consume the same topic (fan-out). - Consumer offset tracking: committed offsets vs. current position. - Use `auto.offset.reset=earliest` for new groups that need to process all history. ## Producer Configuration | Setting | Default | Recommended | |---------|---------|-------------| | `acks` | 1 | `all` (durability) | | `retries` | 0 | 3+ (transient failures) | | `linger.ms` | 0 | 5-100 (batching) | | `batch.size` | 16KB | 32-64KB (throughput) | | `compression.type` | none | `lz4` or `zstd` | | `enable.idempotence` | true | **always true** | ## Consumer Configuration | Setting | Default | Recommended | |---------|---------|-------------| | `auto.offset.reset` | latest | `earliest` (first run) | | `enable.auto.commit` | tru