rag-architecturelisted

# RAG Architecture ## Purpose Design a Retrieval-Augmented Generation pipeline, including document processing, chunking strategy, embedding pipeline, vector database selection, retrieval optimization, and context assembly. ## Scope Constraints Reads source document metadata, query patterns, and infrastructure requirements for pipeline design analysis. Does not execute embedding operations, provision vector databases, or access production data directly. ## Inputs - Source documents (type, volume, update frequency) - Query patterns (user questions, search terms, structured queries) - Quality requirements (relevance threshold, hallucination tolerance) - Latency requirements (real-time, near-real-time, batch) - Cost constraints (embedding costs, storage costs, query costs) ## Input Sanitization No user-provided values are used in commands or file paths. All inputs are treated as read-only analysis targets. ## Procedure ### Progress Checklist - [ ] Step 1: Analyze source documents - [ ] Step 2: Design chunking strategy - [ ] Step 3: Select embedding model - [ ] Step 4: Select vector database - [ ] Step 5: Design retrieval pipeline - [ ] Step 6: Design quality metrics ### Step 1: Analyze Source Documents Understand what's being indexed: - **Document types:** PDFs, web pages, code, structured data, conversations - **Volume:** Number of documents, total size, growth rate - **Update frequency:** Static corpus, daily updates, real-time - **Structure:** Highly structured (ta