eval-runlisted

# Skill: eval-run ## Premise An LLM call is non-deterministic; "it worked when I tried it" is one uncontrolled sample, not a pass. This skill turns a prompt/model/retrieval change into a **measurement**: it runs the versioned eval set through the *current* code and reports the score against a baseline. **Every finding cites the eval case + its score.** A regression claim is `<case-id>` (or `<dataset:row>`) + `<metric> = <score>` + `<baseline score>` + the threshold it crossed. A green "the harness ran" without the per-case scores is not a pass — it is an unread result. A PASS verdict below the baseline is forbidden (see Halt conditions). This is the runner half of the eval loop; the *design* of the dataset + scorers lives in `ai/patterns/evals.md`. ## Adapt to the codebase Detect the project's eval framework and **mirror it** — run the harness the project already has, never impose a second one. Adapt the commands below to the detected tool: | Framework | Detect (grep / files) | Run | Dataset shape | |---|---|---|---| | **promptfoo** | `promptfooconfig.yaml`, `rg promptfoo` | `npx promptfoo eval -c promptfooconfig.yaml --no-cache` | `tests:` YAML with `vars` + `assert` | | **deepeval** | `rg deepeval`, `pip show deepeval` | `deepeval test run tests/` | `LLMTestCase` + `metrics=[...]` in pytest | | **ragas** (RAG) | `rg ragas` | `pytest tests/eval_ragas.py` | HF dataset: `question, contexts, answer, ground_truth` | | **OpenAI evals** | `evals/`, `rg openai.*evals` | `oaie