observability-slolisted

# Observability and Reliability The question this discipline answers is not "is the server up". It is: > **When a user has a bad experience right now, can we tell — and can we find out > why without deploying new code?** A system you cannot debug from its own output is a system you debug by guessing. --- ## The three signals, and what each is for | Signal | Answers | Cost | |---|---|---| | **Metrics** | *Is something wrong?* Aggregate, cheap, alertable | Low, until cardinality explodes | | **Traces** | *Where is it wrong?* One request across every service | Medium, sample it | | **Logs** | *Why is it wrong?* Full detail for one event | High at volume | You need all three, and they must be **linked**: a metric spike leads to an exemplar trace, which leads to the log lines for that request. Three disconnected tools mean three separate investigations and a much longer time-to-diagnosis. Use OpenTelemetry as the instrumentation layer — it keeps the vendor decision reversible, which matters because observability vendors are expensive and switching is otherwise a rewrite. --- ## Structured logging ```json {"ts":"2026-08-24T16:18:12.451Z","level":"error","msg":"payment.capture_failed", "trace_id":"4bf92f...","span_id":"00f067...","request_id":"req_01HQ8", "user_id":"usr_01HQ8","org_id":"org_01HQ8","payment_id":"pay_01HQ8", "provider":"stripe","error_code":"card_declined","duration_ms":842} ``` **Rules** 1. **JSON, not prose.** Grep does not scale; structured queries