eval-iterate

Solid

Iterates on a failing AI/LLM eval (an L2 suite, a golden-set / judge eval, or any eval gating a PR) until it is green AND confirmed, not just luckily passing once. Classifies the failure as a code bug, an eval-definition bug (stale golden item / criteria drift), judge-drift (silent grader version change), or flaky. Applies the minimal fix, then requires N consecutive confirming re-runs — 2 deterministic, 5 for anything a model call grades, since a flat "2" is statistically weak for a stochastic judge. Hard-capped at 5 fix iterations. Refuses to game the eval (Goodhart's Law: skip/delete/overwrite-in-place a case, loosen a threshold) without a second independent check plus `confidence(analysis) >= 90%` and a logged rationale. Composes `ai-engineering`, `confidence`, `critical`, `verify-behavior`. Use when an eval is failing on a PR and needs a real, non-gamed green. Triggers on "this eval is failing", "iterate on this eval", "fix this eval", "get this eval green", "optimize this eval", "/eval-iterate".

AI & Automation 13 stars 2 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 86/100

Stars 20%
38
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Eval Iterate Drive a failing AI/LLM eval to a real green: diagnose, fix, re-run, confirm — capped at 5 iterations, never by weakening the eval. This `SKILL.md` is the **orchestration index**. Load the matching rule file when you need detail — do not preload them. | Phase | Goal | Required rule | | ----- | ---- | ------------- | | 0 | Resolve the target eval + capture the baseline failure | this file | | 1 | Resolve how to run it | this file | | 2 | Classify the failure (verdict required) | [`rules/eval-bug-classification.md`](./rules/eval-bug-classification.md) | | 3 | Apply the minimal fix — gated if it touches the eval itself | [`rules/anti-gaming-guard.md`](./rules/anti-gaming-guard.md) | | 4 | Re-run, then confirm with a second run | [`rules/convergence-confirmation.md`](./rules/convergence-confirmation.md) | | 5 | Iterate or stop at the cap | this file | | 6 | Report (structured exit summary) | this file | Always read [`rules/anti-gaming-guard.md`](./rules/anti-gaming-guard.md) before touching any eval definition (assertion, threshold, golden-set item, judge prompt). The refusals in it apply on every iteration. ## Input The user provides one of: - An eval identifier — an L2 suite name (e.g. `tier-routing`), a golden-set file path, or a test/eval file path. - A PR URL with a failing eval check. - **Nothing** — if `$ARGUMENTS` is empty, auto-detect the failing eval check on the current branch's open PR (see Phase 0). - `--max-iterations <n>` — lowers the cap b...

Details

Author
mthines
Repository
mthines/agent-skills
Created
4 months ago
Last Updated
yesterday
Language
JavaScript
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

eval-loop

Use to run a quality check on any generated artifact, or when the user says "evaluate this", "score this draft", "is this good enough", "run the eval", "review this output", "check it against the rubric". Runs generate, judge, revise, re-judge, approve with the judge in a separate context so nothing grades its own work. Logs every cycle including failures. Writes to workspace/evals/log/. Scores your AI system output against a rubric you wrote. Not for writing that rubric (`eval-spec-authoring`), and not the pre-handoff gate on a deliverable this repo produced, which is `artifact-eval`.

1 Updated 6 days ago
guerrilla2799
AI & Automation Listed

skill-iterate

Serial-iterate every scorable skill autonomously overnight via hill-climb. Default invocation drains all .claude/skills/*/evals-bearing skills (minus an embedded skip-list); per-skill budget capped by 1h wall-clock OR 12 iterations, whichever first. Carries autoresearch's NEVER-STOP-mid-loop discipline. Complementary to /skill-evolve (parallel A/B exploration); /skill-iterate is serial exploitation.

2 Updated today
aberson
AI & Automation Listed

evaluating-ai-output

Evaluate non-deterministic LLM/AI output with evals instead of one-shot "it worked" — define expected behavior first, measure pass@k / pass^k, and grade with code / model / human graders. Use when building or changing an AI/LLM feature, an agent, a prompt, a RAG pipeline, or a classifier, where a single good run is not proof of correctness. Complements writing-tests (deterministic logic) and developing-features-tdd.

5 Updated today
Cristhianzl