← ClaudeAtlas

flaky-test-diagnoserlisted

Diagnose a test that passes and fails without relevant code changes by reproducing the instability, classifying its trigger, and isolating the shared state, timing, ordering, concurrency, randomness, or environment dependency behind it. Use when CI retries hide failures, a test fails only in the suite, or a failure cannot be reproduced reliably on one machine.
imtiazrayhan/agentscamp-library · ★ 1 · AI & Automation · score 77
Install: claude install-skill imtiazrayhan/agentscamp-library
Find the condition that changes a test outcome without guessing or accepting retries as the end state. ## Workflow 1. **Capture the failure signature.** Record the test name, assertion or exception, runner, seed, worker count, duration, environment, retry number, and nearby logs. Separate multiple signatures before investigating. 2. **Measure the baseline.** Run the narrowest failing test repeatedly with the same seed and environment. Report run count and failure rate; do not call a test stable after one passing run. 3. **Vary one dimension at a time.** Compare isolated versus full-suite, serial versus parallel, fixed versus random order, cold versus warm process, local versus CI-like settings, and controlled versus real time. Preserve every command and result. 4. **Classify the trigger.** Check for leaked global state, incomplete cleanup, fixed ports, shared files or records, mutable fixtures, clock and timezone assumptions, unseeded randomness, eventual consistency, unordered collections, resource exhaustion, and true concurrency races. 5. **Find the minimal interference.** Bisect the preceding test set or worker configuration when order matters. Identify the smallest predecessor, shared resource, timing window, or environment variable that changes the outcome. 6. **Distinguish harness defect from product defect.** Do not add waits or mocks until deciding whether the system violates a real invariant under a valid schedule. A race revealed by a test is not automatically a