evaluate-scenarios

Solid

Measure framework overhead by decomposing scenarios into clean-context forks and reporting per-fork load plus aggregate cost.

AI & Automation 81 stars 10 forks Updated yesterday CC-BY-4.0

Install

View on GitHub

Quality Score: 89/100

Stars 20%
64
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

## EXECUTE NOW **Target: $ARGUMENTS** Parse immediately: - empty → evaluate all scenarios in `tests/scenarios/` - a scenario name → evaluate only that scenario - `compare` → evaluate all and compare against a previous run if one exists This harness measures **operational overhead**: the framework instructions an agent must read on top of the task's own content. The unit is the **fork**, not the operation — every `cp-skill-*` runs `context: fork`, so each fork pays its overhead from a fresh context. See `kb/notes/feasibility-is-the-heaviest-forks-net-load.md` for the model. ### 1. Discover scenario files ```bash ls tests/scenarios/*.md ``` Read each scenario. Each has a `## Forks` section with one subsection per fork; each fork has a table of loads: `load | kind | source | hops`, where `kind` is `overhead`, `content`, or `spared`. ### 2. Config (override via $ARGUMENTS, e.g. `notesize=3000 candidates=4 budget=50000 agents_per_fork=on`) | Knob | Default | Meaning | |---|---|---| | `notesize` | 2,000 B | average note/body read | | `candidates` | 3 | content notes opened where a fork prospects bodies | | `spared_bodies` | 3 | bodies an index or description-listing read lets a fork skip | | `index_size` | 3,000 B | one curated index read or scoped description listing | | `validate_out` | 500 B | bytes a `commonplace-validate` run returns into context | | `budget` | 50,000 B | usable-window soft ceiling for the feasibility flag (overhead + content + room to reason) | | `age...

Details

Author
zby
Repository
zby/commonplace
Created
5 months ago
Last Updated
yesterday
Language
Python
License
CC-BY-4.0

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

eval-runner

Run eval scenarios to benchmark Mycelium effectiveness. Execute tasks using reflexion loop, validate against success criteria, record metrics.

41 Updated today
haabe
AI & Automation Listed

scenario

Generate comprehensive edge cases and test scenarios by decomposing a feature or file across 12 risk dimensions. Use for pre-implementation risk discovery, QA planning, regression design, and exhaustive edge-case enumeration. Triggers: 'edge cases for X', 'what could break', 'test scenarios', 'QA plan', 'risk discovery', 'enumerate failure modes'.

2 Updated today
vanducng
AI & Automation Listed

context-budget-auditor

Measures token consumption across every loadable pmo-platform component in a Claude Code / Cowork session — the SKILL.md catalog, core/rules/, core/standards/schemas/specs/disciplines, agents, hooks, and the CLAUDE.md context-file chain — and reports total + per-component estimated token cost, a flagged-bloat section with the applied thresholds stated inline, and a K1/K2 load-vs-reference hint per component. Single Measure mode. Estimates are a heuristic, trend-over-time signal (not falsely-precise absolutes), and the skill roster is sourced live, never hardcoded. Read-and-report only — measures cost, changes nothing. Triggers: "audit the context budget", "measure token consumption", "how much context does the platform load", "what's the per-component token cost", "flag context bloat", "which components are bloated", "context budget report".

0 Updated today
cody-hutson