root-causelisted

# Root Cause Analysis Structured approach to debugging production issues and tracing root causes. Replace guesswork with evidence-driven investigation. ## Investigation Workflow ### 1. Triage - What is the **symptom**? (error message, alert, user report) - What is the **blast radius**? (single user, all users, specific feature) - When did it **start**? Correlate with deployments, config changes, or external events. - Is it **reproducible**? If yes, get a minimal reproduction. ### 2. Gather Evidence Start broad, then narrow: - **Logs**: search for error codes, stack traces, correlated request IDs - **Metrics**: latency spikes, error rate changes, saturation signals - **Traces**: request path through services, slow spans, failed spans - **Recent changes**: git log, deployment history, feature flag toggles ### 3. Form Hypotheses Write down 2-3 possible root causes ranked by likelihood: ``` H1: Deploy of PR #1234 changed the API contract (high — timing matches) H2: Database connection pool exhausted (medium — latency spike correlates) H3: External API rate limit hit (low — no other services affected) ``` ### 4. Test Each Hypothesis Gather evidence that confirms or rules out each one: - **H1**: Check diff of PR #1234. Roll back the change. Does the error stop? Yes → confirmed. - **H2**: Check DB connection pool metrics. If pool size < active connections → confirmed. - **H3**: Check external API response codes. If 429s → confirmed. ### 5. Fix and Verify - Apply the f