build-eval-setlisted

# Build an Eval Set The dataset is the product. A perfect harness over an unrepresentative dataset measures nothing. ## When this applies - An LLM system exists with no measurement - About to change a prompt, model, or retrieval config - Setting up CI gates for AI quality ## Do not use for - Retrieval-specific eval sets → `../../../RAG System/skills/build-rag-evalset` - Making a judge agree with humans → `calibrate-judge` ## Inputs to collect first | Input | Why needed | Default if unspecified | |---|---|---| | The system and what "correct" means | Defines the scorers | **Blocking** | | Real usage data (logs, tickets, transcripts) | The only representative source | Synthesise, and flag the limitation loudly | | Known failure cases | The highest-value cases in the set | Ask — everyone has three | | Which decisions this eval will gate | Sets the required precision | Ask | ## Procedure ### Step 1 — Start from real failures Before writing a single synthetic case, collect every known failure: bug reports, complaints, cases someone had to fix by hand. These are the highest-signal cases in the set and they are free. **Stop condition:** every known failure is a case. ### Step 2 — Sample real inputs Pull from production logs. Stratify: frequent cases, rare cases, long inputs, short inputs, edge formats. If there is no production yet, use the closest real corpus and say so — do not pretend synthetic data is representative. **Stop condition:** ≥ 60% of cases derive from re