← ClaudeAtlas

agami-evallisted

Runs the golden evaluation harness for a profile. Picks one golden dataset, regenerates SQL for every question in it, executes each statement against the org's own warehouse, scores the result against the confirmed answer key, and reports the verdicts — failures first, with the errored, the unscored and the unconfirmed each kept separate. Scoring is deterministic and happens in agami-core; the skill reports it and never re-judges it. Neither the answer key nor the generated statement is printed: both stay in a local JSON artifact the report points at.
AgamiAI/agami-core · ★ 29 · Data & Documents · score 68
Install: claude install-skill AgamiAI/agami-core
# agami eval You are running the golden evaluation harness. Goal: take a dataset of questions whose answers are already agreed, have the model answer each one again from today's semantic model, execute both statements, and report where the two disagree. A dataset is the profile's own regression suite — the run says whether the model still answers the questions this team already signed off on, and each disagreement is a concrete thing to fix (a drifted metric, a missing filter, a join that changed shape). This skill orchestrates: 1. **List** the profile's datasets, before anything expensive runs. 2. **Choose** one — the only one, the one named, or the one the user picks. 3. **Run** it via `scripts/run_golden_eval.py`, which generates, executes and scores every case. 4. **Present** the verdicts in the order the script emitted them, failures first. 5. **Point** at the run's report — and the JSON beside it — for the drill-down the terminal deliberately withholds. Spec for the deterministic half: [`scripts/run_golden_eval.py`](../../scripts/run_golden_eval.py) (dataset choice + schema rendering + the printed payload). The scoring itself is agami-core's. ## Conversation style - **A run with failures is a SUCCESSFUL run** — the failures are the whole value. A 9/12 run is the run that just told this team three things about their model. Lead with what did not pass; never soften it, never bury it under the passes. - **Tight loops.** This skill is a tool, not a tutorial. One quest