spark-guidelineslisted

Use when designing, building, or reviewing Apache Spark 4 data pipelines — batch and streaming DataFrame transforms, lakehouse sinks (Delta/Iceberg), Spark Connect, Structured Streaming (watermarks, checkpoints, transformWithState, state data source), partitioning/AQE/shuffle optimization, and idempotent MERGE/CDC patterns. Reach for this whenever the user mentions Spark, PySpark, Structured Streaming, lakehouse ETL, or distributed dataframe pipelines even if they do not name Spark 4 explicitly.
stevencarpenter/agents · ★ 1 · Code & Development · score 74

Install: claude install-skill stevencarpenter/agents

# Spark Guidelines Shared Spark 4 rubric for agents building expert-level data pipelines. Vendor-general — applies to open-source Spark, Databricks Runtime 16.x+, EMR, or any Spark 4 cluster. Pair with `spark-scala-guidelines` or `spark-pyspark-guidelines` for language-specific idioms. For medallion layering, catalog governance, and cross-engine lakehouse design, also apply `data-engineering-guidelines`. ## Source Of Truth - Apache Spark 4.x docs: https://spark.apache.org/docs/latest/ - Structured Streaming guide (especially `transformWithState`): https://spark.apache.org/docs/latest/streaming/ - The cluster's exact Spark/Delta/Iceberg versions — semantics change between releases - The live catalog and existing table formats in the repo — match what's there ## Spark 4 Baseline - **Assume Spark 4** unless the repo pins an older version. Key 4.x capabilities to prefer over legacy APIs: - **`transformWithState`** (Scala/Java/Python) replaces `mapGroupsWithState`, `flatMapGroupsWithState`, and `applyInPandasWithState` for arbitrary stateful streaming - **Composite state types** — `ValueState`, `ListState`, `MapState` with TTL, timers, initial state, and Avro-backed schema evolution - **State Store Data Source** — read streaming operator state as a DataFrame for debugging (`stateVarName`, flattened vs non-flattened) - **Spark Connect** — client-server mode with high feature parity; toggle via `spark.api.mode` during migration - **Declarative first.** Express transform