ai-evaluationlisted

**CONSUMES**: User evaluation task + target agent/LLM description + optional existing eval configs **PRODUCES**: Applied evaluation judgment rules + benchmark configs + regression baselines + safety audit results + CI/CD pipeline configs + calibrated rubrics # AI Evaluation Capability Pack **Version**: 0.1.0 **Compatibility**: Claude Code (Phase 1); Codex / Cursor / Gemini in Phase 3 **License**: Apache 2.0 --- ## What This Pack Does AI agents set up evaluation by copying tutorial configs. They run promptfoo once with default assertions. They skip statistical rigor — declaring a winner from n=20. They use the same model as judge and generator, hiding self-enhancement bias behind high scores. They never build regression baselines, so prompt changes break production silently. This pack embeds the judgment rules that evaluation engineers apply automatically — rules from real evaluation frameworks, red-team tooling documentation, and statistical testing literature. **Pack = evaluation judgment. Your workflow system = process constraints. No overlap.** --- ## Cross-Cutting Rule: Judge ≠ Optimizer > **When comparing or optimizing LLM outputs, the judge model MUST be a different model family from the generator.** Self-enhancement bias is one of three documented judge failure modes (position, verbosity, self-enhancement) named in the MT-Bench paper (Zheng et al., arXiv:2306.05685). If forced to use the same family, document the bias explicitly and flag results as "internall