adversarial-testinglisted

# Adversarial Testing Normal tests ask "does it do what we meant?" Adversarial tests ask "what did we fail to imagine?" Failures cluster in the input regions nobody modeled, so a suite built from expected inputs is structurally blind to them — not because it is incomplete, but because it was drawn from the wrong distribution. This skill covers classical software and AI systems together, because they are the same problem at different severities. Both come down to one question, and that question routes the entire method. ## The spine: what is your oracle? A *test oracle* is whatever tells you an output is wrong. Everything below follows from how much of one you have. | Oracle situation | What you can do | Layer | |---|---|---| | **Decidable** — you can compute the expected output | Assert it directly. Adversarial value is at input-space edges only | Fuzz for crashes | | **Undecidable, deterministic** — right answer is intractable to state, but the system is repeatable | Assert *relations between runs* instead of values | Metamorphic + differential | | **Undecidable, probabilistic** — no fixed right answer, output varies run to run | Abandon per-case assertions. Measure rates over a corpus | AI red teaming | | **Crash-only** — any input, one universal wrong answer | Memory-safety violation *is* the oracle | Coverage-guided fuzzing | Weyuker formalized the oracle problem in 1982, and it is the reason this skill exists. Where you cannot say what a single correct answer is, y