observability-designerlisted

# Design production-ready observability Instrument services so operators know what is broken, why, and where — before users notice. Three pillars, golden signals, low-noise alerting. ## The three pillars | Pillar | Purpose | Key decision | |--------|---------|-------------| | **Metrics** | Rate, latency, saturation at a glance | RED method for services; USE method for resources | | **Logs** | Structured event trail with correlation IDs | JSON, log-level discipline, sample high-volume streams | | **Traces** | End-to-end request flow across services | Meaningful span boundaries; tail-based sampling for slow/erroring requests | Golden signals to define first: **latency, traffic, errors, saturation** — cover these before anything else. ## Steps 1. **Read the lore first.** `search_lore` for existing observability decisions (dashboards, alert channels, on-call runbooks, APM tooling). Extend in place; don't duplicate. 2. **Identify the service contract.** What does this service promise users? That contract → the SLIs. Route SLO/error-budget design to `slo-architect`. 3. **Design dashboards.** Overview → service → component drill-down. Max 7±2 panels per screen; colour semantics (red = critical, amber = warning, green = healthy); SLO target reference lines. 4. **Define alert thresholds.** Prefer symptom-based over cause-based alerts. Require every alert to have: condition, severity, runbook link, on-call owner. Suppress during known maintenance. 5. **Reduce noise.** Deduplicate