← ClaudeAtlas

run-sefi-benchmarklisted

Operate this repository's real-token-spend blinded paired A/B benchmark -- the full sefi-agents chain versus one strong model on frozen cases, needing explicit per-invocation human authorization of its dollar spend before any model call. Use only when a human explicitly asks to run the sefi benchmark, never for routine retro or efficiency review, and never from a loop.
xsefirosus/sefi-agents · ★ 1 · AI & Automation · score 70
Install: claude install-skill xsefirosus/sefi-agents
# Run Sefi Benchmark Operate the blinded paired A/B harness that compares the full sefi-agents chain against one strong model on a frozen set of cases. The design of record and the trial-record schema live in `benchmarks/README.md`; this skill is the run-and-report workflow. User instructions always override this skill. All factual output follows the anti-hallucination skill: cite or mark UNKNOWN, never guess. Missing telemetry is `null`, never `0`. ## Cadence -- never inside a loop Run this per release, or monthly, or when a human explicitly asks. It is manual and out-of-loop by construction. **Never more than one benchmark run per day, AND only with explicit per-invocation authorization** -- there is no standing pre-authorization, a day passing does not auto-approve a run, and a prior approval never carries to the next run. Same statement in `docs/BUDGET.md` and `docs/METRICS-PROVENANCE.md`. A `loops/*.loop.md` spec MUST NOT call this skill: a benchmark run spends real tokens and time well past `per_run_usd_cap`, and an unattended loop has no authority to approve that. If a loop needs benchmark evidence, it parks a request in `inbox/` for a human. Nothing mechanically prevents a loop from invoking this skill -- there is no runtime guard that blocks a `loops/*.loop.md` from naming it. The "no loop may call this" rule is model-compliance prose, backed by the explicit-per-invocation-authorization gate below (step 2), which a compliant operator will not clear for an unatte