← ClaudeAtlas

eval-enginelisted

Iterate-stage skill: turns a spec into the complete runnable verification layer — binary gates, anchored rubric, paste-ready judge prompt, and harness instructions — in one pass. Use when a feature needs its full eval built — 'create an eval for this spec', 'build the verification layer', 'spec to eval harness', 'turn this PRD into something we can run outputs through' — or when /pm routes such a request here. Do NOT use for the gates+rubric design artifact alone (prd-to-eval), for judge prompts over existing criteria (llm-as-judge-designer), for executing an eval over outputs, or for eval definitions.
Abhillashjadhav/PM-agent-OS · ★ 1 · AI & Automation · score 60
Install: claude install-skill Abhillashjadhav/PM-agent-OS
# Eval Engine Spec in, a runnable verification layer out: gates, rubric, judge prompt, harness — with the one law enforced throughout: disqualifiers gate, tradeables score, never mixed. ## Verification gates (defined first; output is blocked until all pass) - **G1 — Never mixed:** every requirement lands as either a binary gate (disqualifier — fabrication, missing required elements, compliance misses) or a scored rubric dimension (tradeable — clarity, economy, tone). A disqualifier with a 1-5 score or a taste-call as a gate fails. Execution order stated in the deliverable: gates first, any failure = automatic fail, never averaged. - **G2 — Everything operational:** every gate has a yes/no check procedure tagged `mechanical` or `judge`; every rubric dimension has concrete 1- and 5-anchors; the judge prompt embeds them and demands quoted evidence; the harness instructions say who runs what, in what order, and what triggers recalibration. - **G3 — Spec-traced:** every gate and dimension cites its spec phrase. Missing-but-obvious requirements are proposed as spec additions, never silently added. ## Steps 1. **Harvest and sort.** Split compound requirements; run the ship-it test on each (*fails this, aces everything else — ship?* No → gate; tradeable → rubric). Compliance-critical items ("must not miss X") always gate, with a seeded golden case as their check. 2. **Write the gates (3–6):** name · binary check procedure · mechanical/judge tag · why-disqualifying. Vague disqual