← ClaudeAtlas

runtime-validationlisted

Catch the bugs a green test suite misses by observing REAL runtime behavior — tail application logs during test/server runs, validate API responses against their schema contracts, and assert data-integrity invariants (density ≤ 1.0, %≤100, referential integrity). Use when unit tests pass 100% but the running app still errors, when a Pydantic/Zod/serde validation error or schema drift shows up only at runtime, when reviewing whether a test suite actually exercises the running system, or when someone says "all tests pass but it's broken", "smoke test", "the mocks lied", or "verify it works live".
nxtg-ai/forge-plugin · ★ 5 · AI & Automation · score 73
Install: claude install-skill nxtg-ai/forge-plugin
# Runtime Validation Unit tests assert on **mocked inputs and isolated functions**. Runtime validation asserts on the **actual running system** — logs, real HTTP responses, and cross-layer data. A suite can be 100% green while the product is broken, because the mocks never hit the code path that fails. This skill is the discipline that closes that gap. ## Why this exists (real forge incident, not hypothetical) `hooks/scripts/smoke-test-reminder.sh` in this plugin carries the origin verbatim: > On 2026-02-06, an agent spent an entire day writing **2326 mock unit tests** while the > actual product was broken. A duplicate placeholder route intercepted all real requests. > Nobody caught it because **nobody started the actual server**. That is the whole thesis: *a passing mock suite is not evidence the running system works.* The three techniques below are how you get that evidence. ## The three techniques | Technique | What it observes | Catches | |-----------|------------------|---------| | **Log monitoring** | app logs during a test/server run | validation errors, stack traces, swallowed exceptions the assertion never saw | | **Contract validation** | real HTTP responses vs their schema | schema drift, missing/renamed fields, type mismatches between layers | | **Invariant checking** | derived data values | math/domain violations (density > 1.0, negative counts, % > 100) | Runtime validation *composes with* your existing tests — it does not replace them. Run the unit suit