measure-agent-task-successlisted

# Measure agent task success A chatbot is judged per response; an *agent* is judged on whether it finished the job. An agent can produce great-looking individual steps and still fail the task (wrong final state, gave up, looped, took a destructive action). Measure at the task level. ## Score the outcome first, then the path **Outcome (did it succeed?)** - the primary metric: - **Task success rate:** did the agent reach the correct end state? Define "success" concretely per task type (the ticket was resolved, the code passed tests, the correct record was updated, the right answer with the right action taken). - Prefer a **checkable** success signal where possible (tests pass, DB in expected state, API returned expected result) over a judged one. For open-ended tasks, use an LLM-as-judge against a rubric, calibrated. **Path (how well did it get there?)** - the diagnostic metrics: - **Steps / tool calls to completion** (efficiency; blowups mean loops or confusion). - **Tool-call success rate** (did tools get called with valid args and succeed?). - **Cost + latency per task** (rolled up across all the agent's calls). - **Recovery:** did it recover from a tool error, or fail/loop? ## Build the eval 1. **A task dataset**, not a prompt dataset: each item is a full task with a defined success condition (see `build-eval-dataset`). Pull real tasks from production traces (`trace-based-testing`). 2. **Run the agent end to end** in a sandbox; score outcome (checkable or judged) + pa