trace-based-testinglisted

Use this to turn real production traces into a regression test suite, so bugs you saw once never come back. Trigger on "turn traces into tests", "capture this bug as a test", "build tests from production data", "my eval set is out of date", or after debugging a production incident. This is the loop that connects observability to evaluation.
ContextJet-ai/awesome-llm-observability · ★ 26 · Testing & QA · score 72

Install: claude install-skill ContextJet-ai/awesome-llm-observability

# Trace-based testing Observability records what really happened; evaluation checks whether it was good. Trace-based testing is the bridge: promote real production traces into repeatable test cases. Done consistently, your eval set stays representative on its own and every incident makes the app more robust. ## The loop 1. **A trace shows a problem** (a bad answer, a hallucination, a tool misfire, a slow/expensive run). You find it via alerts (`set-up-drift-alerts`) or debugging (`debug-agent-from-traces`). 2. **Capture the trace's input as a test case.** The input, the retrieved context, the expected-correct output (write the gold answer, or a rubric). Redact PII first (`redact-pii-for-tracing`). 3. **Add it to the eval set** (`build-eval-dataset`), tagged with the incident so you know why it exists. 4. **Fix the bug**, confirm the new case passes. 5. **CI runs it forever** (`eval-driven-development`) so the bug cannot silently return. ## Make it systematic, not manual - **Sample continuously**, not just on incidents. Periodically pull a stratified sample of production traces (by topic, difficulty, outcome) and fold them into the eval set so it tracks the real input distribution as it drifts. - **Auto-flag candidates**: traces with low online-eval scores, thumbs-down feedback, errors, or high cost are prime candidates to promote to test cases. - **Keep a "replay" path**: re-run a captured input through the current app and diff against the recorded/expected output. Most