llm-evaluation

# LLM Evaluation ## Purpose Know whether an LLM feature is getting better or worse. Without evaluation, every prompt change is a guess, and the confidence that a change helped is indistinguishable from the confidence that it did not. ## When to Use - Before iterating on any prompt or model in production. - Comparing models, prompts, or retrieval strategies. - Setting up regression testing for an LLM feature. - Deciding whether a quality complaint is real or anecdotal. ## Capabilities - Evaluation-set construction from real usage. - Metric selection: exact match, similarity, rubric-based, task-specific. - LLM-as-judge, with the controls that make it trustworthy. - Regression testing in CI. - Online evaluation and production monitoring. ## Inputs - Real inputs from actual usage, not invented ones. - A definition of correct — which is the hard part. - The current behavior, as a baseline. ## Outputs - An evaluation set that includes the hard cases. - A metric that correlates with what users actually care about. - A baseline score, and a gate that catches regressions. ## Workflow 1. **Build the set from real usage** — Fifty to two hundred real inputs, including the failures. An evaluation set of invented examples measures your imagination, not the system. 2. **Define correct precisely** — For extraction, the exact expected output. For open-ended generation, a rubric with concrete criteria. "A good summary" is not a criterion; "mentions all three decisions and no facts ...