running-cluster-experimentslisted

# Running Cluster Experiments ## Overview This is the **methodology** of experiment campaigns on shared clusters. For the mechanics — MFA, account strings, gres names, quotas, partitions, rsync flags — **REQUIRED BACKGROUND: use `using-drac-clusters`.** **Core principle: a scheduler will kill your job at an arbitrary point, and a cluster will hand you less than you asked for. Design for both, or your allocation buys nothing.** Every rule below is a consequence. The expensive failures are never "the code crashed". They are: the job ran perfectly and wrote nothing; the job ran a different experiment than its name says; the allocation was spent before the first useful instruction. ## Never compute on a login node — use an interactive allocation Login nodes are shared by hundreds of people and staff kill offenders. `squeue`, `ls`, `sacct`, `diskusage_report`, editing, `sbatch`, file transfer: fine. **Anything that loops over data, parses a large file, trains, plots, or runs for more than a few seconds: not fine** — including "just this once to check something", and including work you started before realising how big it was. There is always an alternative, so there is never an excuse. **Interactive compute is a normal allocation, not a special favour:** ```bash salloc --account=<alloc> --cpus-per-task=4 --mem=16G --time=1:00:00 # shell on a compute node srun --jobid=<id> --pty bash # shell inside a running job ``` Short interacti