agent-evaluation-designlisted

# Agent Evaluation Design Turn an AI-reliability question into a decision-ready evaluation contract. Read [the evaluation contract reference](references/evaluation-contract.md) when defining the case set, graders, or acceptance rule. ## Preconditions Identify the system version, decision owner, intended users, operating environment, available evidence, and decision the evaluation must support. If the target behavior or decision is not defined, return the missing decisions before inventing metrics. ## Workflow 1. State the evaluation question and the change, configuration, or baseline being compared. 2. Define the population of real tasks without embedding private inputs. Partition representative, boundary, adversarial, abstention, recovery, and known-failure cases. 3. Pin the system configuration: model or agent version, instructions, tools, context policy, sampling settings, dependencies, and environment. 4. Define a baseline that makes improvement or regression interpretable. Use the current system, a simpler workflow, human performance, or a justified no-system baseline. 5. Assign an oracle or grader to each behavior. Prefer deterministic checks for objective facts; define calibrated human review or rubric-based grading where judgment is unavoidable. 6. Create an error taxonomy that separates wrong answers, missing required content, unsupported claims, unsafe actions, routing failures, refusals, tool failures, and evidence gaps. 7. Define metrics, units, aggregation,