instruction-evallisted

Change one condition an agent runs under (instruction text, the reference material instructions point at, MCP tools and permissions, hooks, skill files), then run the same prompts before and after, several times each, to see what actually changes. Produces an HTML report showing both arms' answers side by side. Use when deciding whether to add a rule, when you need evidence that an existing rule can be deleted, when picking between two wordings of the same rule, or when checking whether material you placed is actually being read. Triggers on 'does this rule do anything', 'compare before and after adding this', 'can I drop this', 'difference with and without this', 'run an A/B'. Not for application code changes or anything automated tests already verify. Built on Claude Code: it spawns `claude -p` subprocesses to run both arms.
2ykwang/agent-skills · ★ 19 · AI & Automation · score 80

Install: claude install-skill 2ykwang/agent-skills

# Condition A/B Change one condition an agent runs under, run the same prompts before and after, and show the difference. The conditions surrounding an agent have no verification. You can read code and tests will catch a regression, but a few lines added to instructions or a reference doc dropped in a directory only ever get judged on whether they sound reasonable. Even the person who put them there has no idea whether they change behavior. This skill replaces that guess with an observation. This runs on Claude Code. Both arms execute as `claude -p` subprocesses, so the CLI has to be available. ## Who writes what The report holds content from two sources, visually separated in the HTML. Never hand-write what the script produces, since transcribing only introduces errors. | Automatic (scripts) | LLM (`insights.json`) | |---|---| | Experiment setup: n, model, both directory paths | What you changed | | Prompt text | The claim about what changes | | Answer text (every arm, every rep) | Differences observed per prompt | | Metric medians and deltas | Metric interpretation | | Run anomalies (errors, permission denials) | Conclusion | ## Procedure ### 1. Design and build the conditions **Pick the axis first.** What you compare against what determines everything else. Only what the user mentioned is a candidate, and even when it reads as obvious you confirm it in step 2. Anything can be the axis: instruction text, reference material instructions point at, MCP tools and perm