optimizing-parquet-storagelisted

# Optimizing Parquet Storage ## When to use - Parquet/lake reads are slow or scan too much data. - Files are too small (many tiny files) or too large (poor parallelism). - Choosing compression, row-group size, or partition layout. - Do NOT use for warehouse-native storage tuning (use the warehouse skills). ## Workflow ``` - [ ] Target ~128MB-1GB files and ~128MB row groups - [ ] Partition by common filter columns (low/medium cardinality) - [ ] Sort within files by a filter column to tighten min/max pruning - [ ] Pick compression: Snappy (speed) or ZSTD (ratio) - [ ] Compact small files; select only needed columns ``` 1. **Right-size files and row groups.** Aim for ~128MB–1GB files and ~128MB row groups so engines get efficient parallelism and pruning. Tiny files kill performance via per-file overhead. 2. **Partition on filter columns** of low/medium cardinality (date, region). Avoid high-cardinality partitioning (user_id) — it creates millions of tiny files. 3. **Sort within files** by a frequently filtered column so Parquet's per-row-group min/max stats enable predicate pushdown (data skipping). 4. **Compression:** Snappy for hot, latency-sensitive data; ZSTD for better ratio and cheaper storage at similar read speed. 5. **Read fewer columns** — the biggest columnar win is column pruning. ## Patterns **Write well-sized, sorted Parquet (Spark):** ```python (df.sort("ordered_at") # tighten row-group min/max for pushdown .repartit