benchmark-design
FeaturedDesign and calibrate a multi-Case capability Benchmark with repeated independent evaluations and a traceable baseline.
AI & Automation 199 stars
23 forks Updated today Apache-2.0
Install
Quality Score: 93/100
Stars 20%
Recency 20%
Frontmatter 20%
Documentation 15%
Issue Health 10%
License 10%
Description 5%
Skill Content
# Benchmark Design
Create and calibrate a multi-Case Benchmark that discovers a specified Test Agent's capability boundary. Own the public Statements, private Rubrics, Case set, configuration, and final baseline. Do not modify the Test Agent State, launch the Test Agent directly, or score a Case yourself.
## Before you start
Require a Test Agent and the capability to measure. If either is missing, ask the user. A Benchmark run also requires a top-level Session with `run_subagent` and the current Agent must have `agent-evaluation` installed. If the Skill is missing or this Session is already a subagent, stop and ask the user to install the Skill or start a top-level Session. Do not begin a partial Benchmark.
## Boundaries
Access only the explicit Test Agent and Benchmark paths. Do not inspect another Agent, Project configuration files, Agent Evaluator State, Evaluator Workspace, or Evaluator Trace. Consume only each Evaluator's terminal protocol response and the returned Test Session id.
Use the Environment's App Data Dir and the explicit Test Agent id:
```text
PROJECT_DIR = <app_data_dir>
PROJECT_ID = <basename_of_project_dir>
PENGUIN_HOME = <parent_of_project_dir>
TEST_AGENT_DIR = <app_data_dir>/agents/<test_agent_id>
BENCHMARK_DIR = <app_data_dir>/agents/<test_agent_id>/benchmarks/<benchmark_id>
SCOREBOARD = <benchmark_dir>/scoreboard.yaml
```
Derive a semantic Benchmark id when the user does not supply one. Require `agent_state/system_config.yaml`; its top-level `v...
Details
- Author
- Prism-Shadow
- Repository
- Prism-Shadow/penguin-harness
- Created
- 1 weeks ago
- Last Updated
- today
- Language
- TypeScript
- License
- Apache-2.0
Similar Skills
Semantically similar based on skill content — not just same category
AI & Automation Featured
agent-evaluation
Run and score exactly one Benchmark Case run with CLI execution, Trace provenance checks, and private Rubric isolation.
199 Updated today
Prism-Shadow AI & Automation Solid
agent-benchmark
Framework for measuring and tracking agent response quality over time. Detects regressions before they reach production. Use when evaluating agent changes, auditing quality, or establishing performance baselines.
520 Updated 1 weeks ago
vibeeval AI & Automation Featured
agent-benchmark-suite
Agent skill for benchmark-suite - invoke with $agent-benchmark-suite
66,198 Updated today
ruvnet