debugging-production-systemslisted

# Debugging Production Systems ## Overview Core principle: **evidence before fixes, and match reproduction fidelity to the failure.** The most expensive debugging mistake is shipping a fix for the bug you *assumed* you had. Every claim — "it's deployed", "the data is fine" — gets verified against reality, not memory or prior summaries. **REQUIRED BACKGROUND:** Understand systematic-debugging (root-cause before fix, red→green). This skill adds the production/ops layer: servers, containers, CI, third-party APIs, real browsers. ## When to Use - Error text comes from a **third-party UI** (iframe, hosted checkout, embedded widget) — that is *their validation rejecting your input or config*, so the bug is byte-for-byte in what you send or how you're configured. - **Works locally, fails in prod** — the cause is drift: environment, config, data, build, or the provider's side. - Bug persists after a deploy — suspect **what is actually deployed/served**, in that order: container build → served bundle → the user's browser session. - A fix shipped and the error *changed but didn't vanish* — progress (peeled onion layer), treat the new error as a new hypothesis, not a regression. - NOT for: purely local dev bugs, test failures with no deploy step, or errors reproducible and fixable entirely in one local process. ## The Workflow 1. **Triage**: quantify impact, decide mitigate-vs-root-cause, start an incident note with deploy SHAs. 2. **Evidence first, change nothing yet**: capture t