benchmark-design

Featured

Design and calibrate a multi-Case capability Benchmark and establish a traceable Formal Baseline.

Web & Frontend 2,389 stars 254 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 93/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark Design Build a multi-Case Benchmark for one Test Agent, calibrate its difficulty with one Run per Case, and record the selected frozen Pilot as the Formal Baseline. This Skill changes the Benchmark, never the Test Agent. It does not run or score the Test Agent. Delegate every evaluation with `run_subagent`, and tell each worker to use `agent-evaluation`. Stop after the Baseline; do not begin optimization. ## Before you start If the request does not identify a Test Agent, target capability, desired baseline score, and Pilot iteration limit, ask for the missing inputs. When they are already supplied, proceed without asking the user to restate them. Treat the current Agent as the **Builder**. A user-specified evaluation `(provider, model_id)` takes priority; otherwise inherit the current Builder Session's complete `Provider` and `Model ID` from the Environment. Never use a Project default as an implicit evaluation runtime. ## Workflow - A **Pilot** is a one-Run-per-Case evaluation used to improve the Benchmark. Unselected Pilot results never enter the Scoreboard; the selected result becomes the Formal Baseline after Freeze. - **Freeze** means the Benchmark revision and evaluation settings stop changing. - A **Formal Baseline** is the accepted result of the selected complete valid Pilot revision, recorded after that exact revision is frozen on one unchanged Agent State version. Follow this order: 1. Validate the Test Agent, target capability, resolved evaluati...

Details

Author
Prism-Shadow
Repository
Prism-Shadow/penguin-harness
Created
2 months ago
Last Updated
today
Language
TypeScript
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

benchmark-curator

Design, curate, version, and quality-control benchmark or evaluation corpora for Agent Skills and quality workflows, including task taxonomies, discovery/forced/negative controls, adversarial and regression cases, difficulty strata, holdout isolation, provenance, duplication, contamination risk, coverage balance, and immutable benchmark hashes. Use when the user asks to build or maintain an eval dataset, golden set, benchmark suite, regression corpus, holdout, challenge set, or representative test cases. Do not use to execute the model experiment or claim lift (use skill-evaluator), to design the grading rubric (use rubric-designer), to edit the candidate skill (use skill-creator), or to treat leaked/known cases as a clean holdout.

1 Updated today
CometWeb-io
AI & Automation Featured

agent-evaluation

Run one specified Test Agent on one specified Benchmark Case exactly once, privately score that execution, and return one protocol result.

2,389 Updated today
Prism-Shadow
AI & Automation Listed

benchmark-implement

Run an approved PRD through the existing receipt-backed implement workflow, analyze its runtime session with a fresh independent evaluator, emit a deterministic process report, and compare it with an earlier run. Use when the user invokes "$benchmark-implement", asks to benchmark the implement skill or harness, rerun a fixed PRD after harness changes, analyze implementation-session efficiency, or compare baseline and candidate harness behavior.

1 Updated today
yansfil