pycegerb-yodalisted

# PySpark on Iceberg Every claim here is traceable to the Apache Spark or Apache Iceberg docs — links per section. Version-sensitive defaults are marked; verify against the version you run before relying on one. ## Schema and types - **Declare the schema; don't infer it.** `inferSchema` costs an extra pass over the data and pins correctness to whatever the sample happened to contain. Pass an explicit `StructType`. - **Never `select("*")` into a write.** Column order and presence become part of your contract the moment something downstream reads it. - **`toPandas()` and `createDataFrame(pandas_df)` move data through the driver.** The docs are explicit that this should be "done on a small subset of the data", and that not all Spark/Arrow types are supported — an unsupported column type raises. ([Arrow in PySpark](https://spark.apache.org/docs/latest/api/python/tutorial/sql/arrow_pandas.html)) ## Transformations that don't bloat the plan - **Don't chain `withColumn()` in a loop.** The PySpark docstring is explicit: it "introduces a projection internally", so calling it repeatedly "can generate big plans which can cause performance issues and even `StackOverflowException`". Use one `select()` with all the columns, or `withColumns({...})` (Spark 3.3.0+) to add many at once. - **`try_cast` instead of `cast` on untrusted input.** It is "identical to `CAST`, except that it returns `NULL` result instead of throwing an exception on runtime error" — Spark's