ai-eval-harnesslisted

# Purpose Define datasets, metrics, graders, and regression gates for AI systems (product features or this skill portfolio). # When to Use / When NOT to Use **Use when:** LLM/agent features; prompt/skill changes; “are we better?”; offline evals; skill portfolio regression. **Do not use when:** non-AI CRUD with no model calls; pure infra without generative behavior. # Preconditions Clear target behavior and some way to run the system/skill under test (or design-only mode if runtime unavailable). # Inputs / Outputs **Inputs:** feature/skill under test, CONTEXT_PACK, success criteria, existing evals if any. **Outputs:** `EVAL_PLAN` then `EVAL_REPORT` when executed. # Upstream / Downstream **Upstream:** spec-validator, test-strategy-designer, engineering-os, skill authors. **Downstream:** feature-implementer, quality-gate, pr-generator, delivery-planner. # Core Principles 1. Evals need fixed cases + graded outcomes — not vibes alone. 2. Split: unit-ish (deterministic checks), scenario, adversarial/safety, human review sample. 3. Measure regressions; version datasets. 4. Cost/latency are metrics too. 5. For skills: trigger accuracy, artifact validity, stop-condition obedience. 6. Report confidence intervals qualitatively when N is small. 7. Never fabricate scores. # Process ## Modes - `product` — AI feature in the app - `skill` — evaluate an Agent Skill / portfolio path 1. Define behaviors and failure modes to catch. 2. Build/extend dataset cases (inputs, expected