experiment-queue

# Experiment Queue > ⏱ **External cadence: visibility only.** This skill already runs its own > detached server-side scheduler (60s poll + `depends_on` + wave transitions). > Use its status output for overnight visibility (N done / N running / N > pending); do **not** wrap it in a second `/loop` / `CronCreate` poll — that > duplicates the scheduler on an uncoordinated clock and races the > wave-transition logic it was built to prevent. See > [`shared-references/external-cadence.md`](../shared-references/external-cadence.md) > ("don't duplicate an existing scheduler"). Orchestrate large batches of ML experiments on SSH remote GPU servers with proper state tracking, OOM retry, stale cleanup, and wave transitions. ## When to Use This Skill Use when `/run-experiment` is insufficient: - **≥10 jobs** that need batching across GPUs - **Multi-seed sweeps** (e.g., 21 seeds × 12 cells) - **Wave transitions** (run wave 1, wait, run wave 2, wait, run wave 3...) - **Teacher+student chains** (train teacher then distill; auto-trigger student after teacher done) - **OOM-prone configs** where you need to retry with different GPU or wait - **Mixed seed grids** where failed cells need re-running Do NOT use for: - Single ad-hoc experiment (use `/run-experiment`) - Modal/Vast.ai deployments (those have their own orchestration) - Experiments that need manual inspection between runs ## Why This Exists Based on session audit (2026-04-16), the major wall-clock sinks in multi-seed grid experime...