monitor-rag-qualitylisted

# Monitor & improve RAG quality Most "the LLM is wrong" bugs in a RAG app are actually **retrieval** bugs - the model was handed the wrong context and did its best. Measure both halves separately. ## The 4 metrics that matter (RAG quartet) | Metric | Question | Which half | |---|---|---| | **Context precision** | Are the retrieved chunks relevant (not noise)? | Retrieval | | **Context recall** | Did retrieval find *all* the needed info? | Retrieval | | **Faithfulness** | Is the answer grounded in the retrieved context (no made-up facts)? | Generation | | **Answer relevance** | Does the answer actually address the question? | Generation | [Ragas](https://github.com/vibrantlabsai/ragas) implements all four; [DeepEval](https://github.com/confident-ai/deepeval) and most observability platforms (Langfuse, Phoenix, Opik) have RAG evaluators too. ## Diagnose: retrieval vs generation Read a failing trace (see `debug-agent-from-traces`): - **Low context recall/precision** → fix **retrieval**: chunking strategy, embedding model, top-k, reranking, query rewriting, metadata filters. No amount of prompt tuning fixes missing context. - **Good context but low faithfulness** → fix **generation**: prompt the model to answer *only* from context, add a groundedness guardrail, lower temperature, cite sources. - **Good context, hallucinates anyway** → the model is ignoring context: tighten the prompt, or the context is too long and it's getting lost ("lost in the middle"). ## Set it up 1