llm-eval-suite-scaffolderlisted

The hardest part of LLM evaluation is starting. This skill scaffolds a complete, runnable eval suite for a feature — dataset, metrics, baseline, and CI wiring — using the framework that fits the stack (DeepEval for Python/pytest, promptfoo for config-driven CLI, RAGAS for RAG-specific metrics). ## When to use this skill - An LLM feature ships with no evals and you need a gate before changing it further. - You're about to tune a prompt or swap a model and want to measure the change, not guess. - You're adding an LLM feature to CI and need a suite that fails on regressions. ## Instructions 1. **Pin the task and the unit of scoring.** State exactly what the feature must produce and how one output is judged: exact match, JSON-schema valid, a numeric tolerance, or an LLM-as-judge rubric. An ambiguous success criterion is the real bug — resolve it first. 2. **Build a representative dataset.** Collect 20–50 real inputs with expected behavior, deliberately oversampling hard and adversarial cases (empty input, ambiguity, the format that broke last time, the prompt-injection attempt). Freeze it under version control. For RAG, capture the gold passages too. 3. **Pick the few metrics that matter.** Two or three the feature is actually graded on — not every metric the framework offers. Faithfulness and answer relevancy for RAG; task accuracy and format validity for extraction; a calibrated rubric ([llm-as-judge-scorer](/skills/data/llm-as-judge-scorer)) for open-ended output. 4. **Cho