shadow-mode-validationlisted

Design pattern for shadow-mode validation — running a candidate decision mechanism (typically an LLM judgment) in observe-only parallel with the live path, recording what it WOULD have decided per event, and letting the accumulated record decide enforcement. Use when an unvalidated stochastic mechanism is about to replace or filter a live behavior (a one-way door for output quality), when a selector/classifier/gate has no published reliability evidence for the model class in play, or when designing the isolation, kill-switch, and exit criteria for a shadow deployment. NOT for aggregate readings over stored state (that is read-only-instruments), NOT for the audit-log record schema itself (that is replayable-audit-logs — a shadow log IS one of those logs), and NOT a substitute for unit tests — shadow mode validates decision quality in production traffic, not code correctness.
shimo4228/agent-observability-patterns · ★ 1 · AI & Automation · score 78

Install: claude install-skill shimo4228/agent-observability-patterns

# Shadow-Mode Validation **Shadow mode** runs a candidate decision mechanism alongside the live path, records its would-be decision per event, and changes nothing. The name and the stance come from the established deployment lineage — *shadow deployment* / *shadow testing* / *dark launch* in ML and service operations, where a new system receives production traffic whose outputs are recorded but never served. This skill applies that lineage to in-agent decision mechanisms: the live behavior stays exactly as it was; the only output is an append-only record of "here is what the new mechanism would have done." Enforcement — actually letting the mechanism steer behavior — is a separate, later decision made from the accumulated record. Worked example: an agent adds a pass-1 LLM judgment that selects which of its skills apply to the current situation. Instead of wiring the selection into the prompt on day one, the selector runs shadow-only for 2–4 weeks, recording per event what it would have injected; the enforcement decision is then made from hallucination rates, fail-open rates, and realized-reduction distributions in the record. **How it composes with the sibling skills.** The shadow record is a replayable audit log ([`replayable-audit-logs`](../replayable-audit-logs/SKILL.md) owns the schema: b64+sha256 untrusted text, verdict reason codes, no silent fallbacks). The go/no-go reading over that log is an instrument ([`read-only-instruments`](../read-only-instruments/SKILL.md)