← ClaudeAtlas

eval-iteratelisted

Iterates on a failing AI/LLM eval (an L2 suite, a golden-set / judge eval, or any eval gating a PR) until it is green AND confirmed, not just luckily passing once. Classifies the failure as a code bug, an eval-definition bug (stale golden item / criteria drift), judge-drift (silent grader version change), or flaky. Applies the minimal fix, then requires N consecutive confirming re-runs — 2 deterministic, 5 for anything a model call grades, since a flat "2" is statistically weak for a stochastic judge. Hard-capped at 5 fix iterations. Refuses to game the eval (Goodhart's Law: skip/delete/overwrite-in-place a case, loosen a threshold) without a second independent check plus `confidence(analysis) >= 90%` and a logged rationale. Composes `ai-engineering`, `confidence`, `critical`, `verify-behavior`. Use when an eval is failing on a PR and needs a real, non-gamed green. Triggers on "this eval is failing", "iterate on this eval", "fix this eval", "get this eval green", "optimize this eval", "/eval-iterate".
mthines/agent-skills · ★ 13 · AI & Automation · score 79
Install: claude install-skill mthines/agent-skills
# Eval Iterate Drive a failing AI/LLM eval to a real green: diagnose, fix, re-run, confirm — capped at 5 iterations, never by weakening the eval. This `SKILL.md` is the **orchestration index**. Load the matching rule file when you need detail — do not preload them. | Phase | Goal | Required rule | | ----- | ---- | ------------- | | 0 | Resolve the target eval + capture the baseline failure | this file | | 1 | Resolve how to run it | this file | | 2 | Classify the failure (verdict required) | [`rules/eval-bug-classification.md`](./rules/eval-bug-classification.md) | | 3 | Apply the minimal fix — gated if it touches the eval itself | [`rules/anti-gaming-guard.md`](./rules/anti-gaming-guard.md) | | 4 | Re-run, then confirm with a second run | [`rules/convergence-confirmation.md`](./rules/convergence-confirmation.md) | | 5 | Iterate or stop at the cap | this file | | 6 | Report (structured exit summary) | this file | Always read [`rules/anti-gaming-guard.md`](./rules/anti-gaming-guard.md) before touching any eval definition (assertion, threshold, golden-set item, judge prompt). The refusals in it apply on every iteration. ## Input The user provides one of: - An eval identifier — an L2 suite name (e.g. `tier-routing`), a golden-set file path, or a test/eval file path. - A PR URL with a failing eval check. - **Nothing** — if `$ARGUMENTS` is empty, auto-detect the failing eval check on the current branch's open PR (see Phase 0). - `--max-iterations <n>` — lowers the cap b