a-serial-queue-is-a-measurement-you-did-not-take

Solid

Use at study design when the run queue is priced, at implementation when the first launcher script is written, and at every stage boundary while fits, simulations or samples are still running, especially when the brief states a CPU allocation and your queue happens to run one job at a time. Covers why running nproc is not the missing step, a stated allocation versus an enforced cgroup limit, the two-job A/B and the width ladder that settle how many concurrent processes to run, processes versus per-process threads, and re-pricing every arm you declined while the node sat idle.

AI & Automation 805 stars 25 forks Updated 2 weeks ago NOASSERTION

Install

View on GitHub

Quality Score: 82/100

Stars 20%
97
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Running `nproc` is not the missing step. Testing a second job is Two runs of the same task, on the same node class, under the same brief. Both read the sentence *"about 3 CPU cores allocated to you and other agents are working on other tasks on the same node."* **Both ran `nproc` in their first tool call** and both got the same answers: 32 cores, 960 GB of memory, a load average between 0.5 and 1.0 one minute in, and `cat /sys/fs/cgroup/cpu.max` printing nothing at all. One of them then ran its fine-tunes one or two at a time at three threads each for nine hours: 10 trained models, 706 model-minutes over a 498-minute window — **1.4 processes in flight, about four of the thirty-two cores.** Its queue scripts carry the reason in a header comment: *"~3 cores are allocated here and the node is shared."* The other launched its folds five at a time: 57 trained models, 2,173 model-minutes over 537 minutes — **4.0 in flight averaged over the whole window** — and a better graded file. So the diagnosis "they never measured the machine" is wrong, and a skill that told you to measure the machine would have changed nothing. The measurement was taken, written down, and then overruled by a sentence in the brief. **The width of the queue is the decision that never gets an experiment.** ## What a stated allocation is `nproc` reports the host. `len(os.sched_getaffinity(0))` reports the mask you may run on. Only a cgroup reports a limit that is *enforced*: ```bash nproc python3 -c "impo...

Details

Author
tangxiangru
Repository
tangxiangru/AutoR
Created
6 months ago
Last Updated
2 weeks ago
Language
Python
License
NOASSERTION

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

count-your-hypotheses-not-your-arms

Use at study design, implementation, experimentation and analysis when the run has a hard wall clock, models that take hours to train, and more than one configuration you would like to try in parallel - several seeds, several widths, a warm restart, a variant of the variant. Covers reading the machine you actually have instead of the thread count you typed, measuring the contention tax rather than assuming it, the distinct-hypothesis count that decides whether a launch buys anything, computing `converged` from the log instead of declaring it, reporting a blend of arms the clock cut as the repair it is, and the idle-cores failure at the other end of the run.

805 Updated 2 weeks ago
tangxiangru
AI & Automation Listed

running-cluster-experiments

Use when planning, budgeting, submitting or resuming a multi-job experiment campaign on a Slurm/HPC cluster — sizing walltime, shaping jobs and arrays, deciding what to submit first, or after a job hit its walltime, produced no results, silently ran the wrong configuration, or got the wrong number of GPUs.

4 Updated 6 days ago
chgagne
AI & Automation Solid

a-throughput-number-belongs-to-the-runtime-not-the-model

Use at the survey, study design and implementation stages when the task needs a pretrained model, solver or library you must download and run on the machine you were given, and the first configuration you try is too slow to cover the split in the time you have. Covers why the seconds-per-item you just measured is a property of the runtime, the workload and the machine as much as of the component, which field to change before demoting it, re-asking the component question after you fix the runtime, and checking that your fallback still has the property you picked the original for.

805 Updated 2 weeks ago
tangxiangru