spark-pyspark-guidelineslisted

# Spark PySpark Guidelines Language-specific Spark 4 rubric for Python agents. Always apply `spark-guidelines` first for engine-level pipeline design; use this skill for Python idioms. General Python style still applies — full type annotations, pytest over ad-hoc scripts, `pyproject.toml` packaging; for substantial non-Spark Python modules, defer to the python-implementer or python-reviewer agents, which carry the full `python-guidelines` rubric. ## Source Of Truth - PySpark API docs: https://spark.apache.org/docs/latest/api/python/ - Structured Streaming `transformWithStateInPandas` guide - The repo's `pyproject.toml` or `requirements.txt` — PySpark version, Python version, and test runner ## API Choice - **DataFrame API (PySpark SQL)** is the default for all ETL. Do not use **RDD** in new Spark 4 pipeline code. - **Pandas API on Spark (`pyspark.pandas`)** only when the team already standardized on it or for pandas-familiar exploratory transforms on moderate data — production pipelines should prefer the core DataFrame API for Catalyst optimization visibility. - **pandas UDFs** (Arrow-optimized) when vectorized Python logic is unavoidable; plain Python UDFs only as a last resort on cold paths. ## Idiomatic Transforms - Express transforms as **Column** expressions via `F.col`, `F.when`, `F.struct`, etc. Chain with `select`, `withColumn`, `filter`. - Package reusable stages as functions `def clean_orders(df: DataFrame) -> DataFrame` — not copy-pasted notebook cells. - Us