set-up-drift-alertslisted

# Set up drift alerts An LLM app that worked at launch degrades quietly: a model provider updates weights, users start asking different questions, a prompt change trades quality for cost. Drift alerting turns "we found out from an angry customer" into "we got paged when the metric moved." ## The four drifts worth alerting on 1. **Quality drift** - your online eval scores (faithfulness, relevance, hallucination rate) trend down. The one that matters most and the one people forget to watch. Requires online evals (see `add-llm-evals`). 2. **Cost drift** - tokens/request or spend/day creeps up (context bloat, a retry bug, a prompt that grew). Cheap to track, common, expensive to ignore. 3. **Latency / error drift** - p95 latency or error rate rises (provider slowdown, rate limits, a slow retrieval). 4. **Input drift** - production inputs move away from your eval set (new topics, new languages, longer inputs). Your evals stop being representative, so quality can drop *without* your quality metric catching it. Detect via input length/embedding-distribution shift. ## How to set it up 1. **Emit the signals** as span attributes / metrics (see `instrument-llm-observability`): per-request tokens, cost, latency, error, and online eval score. 2. **Baseline them** over a stable window (e.g. last 7-14 days) to know "normal." 3. **Alert on deviation, not absolute values** - page when a metric moves meaningfully vs its own baseline (e.g. quality score down >X%, cost/request up >Y%, error