count-your-hypotheses-not-your-arms

# Six arms of one architecture is one experiment, run six times Launching a training run costs one backgrounded command. Killing one costs a decision. So arms accumulate: a wider net, a warm restart, another seed, a variant of the variant. Each launch is defensible on its own and nothing in the run ever subtracts. The usual advice — *parallel arms divide your cores* — is a guess, and on a shared node it is often the wrong guess in the wrong direction. Do not guess. A run with several live arms owes three ledgers, and it is almost always the third one that costs the score. ## Ledger 1 — the machine you have, not the one you typed `torch.set_num_threads(3)` and `OMP_NUM_THREADS=3` are your choices. They are not a hardware fact, and the moment you write one it starts being quoted back to you as a constraint. Read the real numbers, into a file, before the first launch and again at every stage boundary: ```bash nproc cat /sys/fs/cgroup/cpu.max 2>/dev/null # the quota, if there is one cat /proc/loadavg ``` > **A resource number that appears in your report must appear in a file on disk > first. If it does not, you have quoted your own configuration flag as a > measurement.** The tell is mechanical: grep your notes for the core count you cite and see whether any artifact contains it. Measured on the run below, its own monitoring script wrote `{"nproc": 44, "load_average_1min": 22.24}` every fifteen minutes for three hours, while its report said the model was *"trainable on...