prompt-regression

Solid

Statically checks harness assets against a known-answer probe set after rule/skill changes — inspects the changed source for the trigger phrases, chain links and gate conditions each probe expects, and reports PASS/FAIL/SKIP per probe. Source inspection only; it does not run live sessions, so it catches assets that no longer SAY the right thing, not models that stop DOING it. Triggers on "prompt regression", "did my changes break anything", "regression check", "test harness changes".

AI & Automation 14 stars 1 forks Updated today MIT

Install

View on GitHub

Quality Score: 83/100

Stars 20%
39
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# prompt-regression — Harness Regression Detection After CLAUDE.md edits, rule changes, or new skill additions, harness behavior can silently regress. This skill runs a lightweight probe suite against the changed assets and compares outputs against saved baselines to surface regressions before they reach production. > **Scope distinction** > - harness-doctor: structural completeness (files, links, drift) > - prompt-regression: **behavioral correctness** — did the change alter expected AI response patterns? --- ## Triggers - `/prompt-regression` - "prompt regression", "regression check", "regression test" - "did my rule change break anything", "test harness changes", "verify harness behavior", "make sure my edit didn't change behavior" - After significant CLAUDE.md edits or new skill commits --- ## Execution Steps ### Step 1. Identify Changed Assets ```bash # What changed since last commit (or last N commits) git diff HEAD~1 --name-only -- CLAUDE.md .claude/ plugins/ ``` Classify each changed file: - `CLAUDE.md` → **core behavior** (high impact) - `.claude/rules/*.md` → **rule layer** (medium impact) - `plugins/*/skills/*/SKILL.md` → **skill behavior** (scoped impact) - `plugins/*/skills/*/SKILL.md` (trigger phrases changed) → **trigger routing** (high impact) If no changes detected: report "No harness changes since last commit — regression check skipped." --- ### Step 2. Load Probe Suite Check for custom probes: ```bash ls .claude/regression/probes.md 2>/dev/nul...

Details

Author
chrono-meta
Repository
chrono-meta/forge-harness
Created
3 months ago
Last Updated
today
Language
Shell
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

prompt-regression-tester

Builds and runs comparison suites that catch behavioral drift when a prompt, model, or workflow configuration changes. Use when the user says "did the prompt change regress anything", "compare these prompt versions", "regression-test this prompt change", or "/agent-collab:prompt-regression-tester." Also offer this proactively when someone is about to swap a prompt, model, or tool wiring in a live workflow without a way to check whether existing behavior held.

0 Updated 2 days ago
sumitake
AI & Automation Listed

prompt-audit

Static sweep of every prompt-assembly and output-parsing site for the five prompt-engineering defects — free-text regex/split/JSON.parse where the provider offers a schema, instructions concatenated with untrusted user or retrieved content in one blob, a structured-output call with no schema validation and no repair path, sampling parameters wrong for a single-answer call (temperature > 0 where the provider exposes it — or a temperature set at all where the provider has removed it and a non-default value is a 400), and a prompt literal with no version id feeding logs + cache key + eval run. Emits one finding per site with <path:line> + a real excerpt + the pattern's closure verb. TRIGGER — any diff touching a prompt, a model call, or an output parser; dispatched by /ai-audit and by @ai-feature-reviewer dimension 2. ANTI-TRIGGERS (do NOT fire) — authoring or improving a prompt (that is /add-ai-feature Phase 4); the prompt-injection exploit judgment or the output→sink review (that is @llm-security-reviewer LLM0

1 Updated today
adnanmokhtar
Code & Development Listed

harness-audit

Use when asking whether the harness now does natively what we automate by hand — "audit our rules/skills/hooks", "what's redundant", "does Claude Code do this already", or the monthly check. Verdicts every rule, skill, script, and hook against live harness behavior, then files issues. Advisory only — never edits.

5 Updated today
auerbachb