← ClaudeAtlas

skill-benchlisted

Benchmarks a skill's outcome quality and consistency — same task with the skill available vs masked, k trials per arm, scored on a rubric (pass@k, pass^k). Triggers: 'benchmark this skill', 'run skill-bench', 'does this skill actually help'.
tammai/bigin-skills · ★ 0 · Web & Frontend · score 68
Install: claude install-skill tammai/bigin-skills
# skill-bench Runs the same bench task k times with a skill available and k times with it masked, in isolated git worktrees, and scores each trial against an objective rubric. Never auto-fixes or auto-commits — proposes a report, same discipline `harness-audit` and `sprint-distill` already enforce. **Trial cap:** default 2 skills × 3 tasks × k=3 × 2 arms = 36 trials per invocation. If the requested scope (skills × available tasks × k × 2) would exceed this, state the computed total and ask for explicit confirmation before running any trials. ## When not to use - Project-local only — audits this plugin's own skills, never ships downstream. - Auditing hooks, frontmatter, or docs staleness against the official docs — that's `harness-audit`. ## Phase 0: Select scope and self-heal 1. Target skill(s) — default coverage set is `write-tests`, `task-workflow`, `debug-workflow` (the initial 3-skill set; see `benchmarks/<skill>/tasks.json` for what's actually defined — don't invent tasks for a skill that has no `tasks.json`, report it missing instead). 2. `k` — default 3. 3. **Safety check, always run first:** `node .claude/skills/skill-bench/scripts/mask.mjs status <skill>` for every skill in scope. If any comes back `masked`, a prior run crashed mid-trial — run `unmask` on it before doing anything else. This is the actual mechanism behind "the mask is restored even if a trial errors": idempotent self-healing at the top of every run, not a language-level trap around a subagent ca