ai-eval-plan

# AI Eval Plan Skill You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement. This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test — so a prompt change that quietly makes outputs worse can't ship. ## Required Inputs Ask for these only if they aren't already provided: - **The feature & task** — what the model does and what "good output" means to a user. - **Failure modes that matter** — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow). - **Available data** — any real examples, logs, or labelled cases; or note there are none yet. - **Who judges quality** — automated checks, an LLM judge, human raters, or a mix. - **The decision this gates** — ship/no-ship, model selection, or prompt iteration. ## Output Format ### Eval Plan: [feature] **1. What we're measuring** — the task, and a one-line definition of a good vs. bad response. **2. Eval dataset** - **Cases:** how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set). - **Coverage:** the slices/scenarios that must be represented (edge cases, adversarial, each major input type). - **Golden answers / references:** present or not, and how they were created. **3. Metrics & rubric** - **Per-dimension scores** — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes. - **Automated ...