ai-evals

# AI Evaluation and Fine-Tuning Methodology Skill **Core stance**: an eval is an instrument. An untrusted instrument is worse than no instrument, because it produces confident wrong numbers that ship regressions. Fine-tuning is an optimization loop around that instrument. If the instrument is weak, training just makes the model better at gaming bad measurement. This skill is the cross-domain methodology layer that domain eval and model-lifecycle skills defer to: how to keep an LLM-as-judge honest, integrate eval frameworks, choose between prompting/context/tools/test-time compute/SFT/preference/RFT/PEFT/ distillation, derive thresholds instead of guessing them, and stop flaky runs or training leakage from masquerading as progress. This is the **methodology umbrella** for evals. Domain skills own *what* to measure; this skill owns *whether you can trust the measurement*. - Building an eval system for a **coding agent** -> [ai-coding-agents-observability-evals](../ai-coding-agents-observability-evals/SKILL.md) - Evaluating **RAG / retrieval / search** -> [ai-rag](../ai-rag/SKILL.md) - Running **Hub model benchmarks** (inspect-ai, lighteval) -> use the `huggingface-skills:` plugin (external) - General **LLM lifecycle** decisions -> [ai-llm](../ai-llm/SKILL.md) - This skill: **judge bias, framework choice, calibration, reproducibility, optimization technique gates** — the parts those four share and none owns in depth. ## ASCII Flow ```text eval need | v define verifi...