rseng-big-data-processing

# Big data processing for research "Big" starts where the current tool breaks: the dataset that no longer fits in RAM, the loop over files that no longer finishes overnight. The escalation path matters more than any framework - each step up costs complexity, debuggability and reproducibility, so take the smallest step that works. ## The escalation ladder 1. Optimize in place first: columnar formats with predicate pushdown (Parquet - rseng-scientific-file-formats), dtype downcasting, reading only needed columns; profile before scaling (rseng-performance-profiling) - many "big data" problems are memory-layout problems. 2. Out-of-core on one machine: chunked iteration (read-process- write per chunk), memory-mapped arrays, or Dask's lazy collections on a single node. One machine with streaming processing handles far more than intuition suggests, with none of the distributed complexity. 3. Embarrassingly parallel batch: independent per-file/per-chunk jobs as cluster job arrays (rseng-hpc-computing) or a workflow engine (rseng-workflows) - the RIGHT answer for most research sweeps, and simpler than any framework. 4. Distributed frameworks: Dask (scales NumPy/pandas/xarray idioms; native in the Pangeo geoscience stack) or Spark (SQL-flavored tabular pipelines, industry-standard cluster tooling) when computation genuinely needs cross-partition coordination: shuffles, joins, global aggregations over larger-than-node data. Skipping strai...