streaming-datalisted

# Streaming Data Processing Build production-ready event streaming systems and real-time data pipelines using modern message brokers and stream processors. ## When to Use This Skill Use this skill when: - Building event-driven architectures and microservices communication - Processing real-time analytics, monitoring, or alerting systems - Implementing data integration pipelines (CDC, ETL/ELT) - Creating log or metrics aggregation systems - Developing IoT platforms or high-frequency trading systems ## Core Concepts ### Message Brokers vs Stream Processors **Message Brokers** (Kafka, Pulsar, Redpanda): - Store and distribute event streams - Provide durability, replay capability, partitioning - Handle producer/consumer coordination **Stream Processors** (Flink, Spark, Kafka Streams): - Transform and aggregate streaming data - Provide windowing, joins, stateful operations - Execute complex event processing (CEP) ### Delivery Guarantees **At-Most-Once**: - Messages may be lost, no duplicates - Lowest overhead - Use for: Metrics, logs where loss is acceptable **At-Least-Once**: - Messages never lost, may have duplicates - Moderate overhead, requires idempotent consumers - Use for: Most applications (default choice) **Exactly-Once**: - Messages never lost or duplicated - Highest overhead, requires transactional processing - Use for: Financial transactions, critical state updates ## Quick Start Guide ### Step 1: Choose a Message Broker See references/broker-selection.md