← ClaudeAtlas

dense-retrieval-usagelisted

This skill should be used when indexing or searching a corpus with the turbovec dense ANN retriever, choosing quantized embedding-based retrieval over lexical matching, handling the RuntimeError raised when the turbovec + local extras are not installed, or when keyword search misses paraphrases and synonyms (vocabulary-mismatch problems).
josix/agentic-retrieval · ★ 0 · AI & Automation · score 72
Install: claude install-skill josix/agentic-retrieval
# Dense Retrieval Usage (turbovec) Dense approximate-nearest-neighbor (ANN) retrieval over quantized embeddings, via `TurbovecRetriever` (`retrieval/retrievers.py`), backed by [turbovec](https://github.com/RyanCodrai/turbovec)'s TurboQuant quantizer. ## What it is `TurbovecRetriever` embeds each document with a `sentence-transformers` model (default `all-MiniLM-L6-v2`, d=384), builds a `TurboQuantIndex` over the resulting vectors, and ranks by inner-product similarity on the quantized vectors. TurboQuant is a **data-oblivious** quantizer — a fixed random rotation plus per-coordinate calibration, derived from math rather than learned from the corpus, so there is **no training phase and no rebuild** as the corpus grows (`index.add(vectors)` is enough). It compresses embeddings to 2-4 bits/dimension (up to 16x smaller than float32) while its length- renormalized scoring keeps inner-product estimates unbiased at zero search-time cost. ## When to use it (vs lexical / pi-serini) **Strengths:** - Matches on **meaning**, not shared tokens — bridges vocabulary mismatch between query and document wording (the failure mode plain lexical retrieval cannot fix without enrichment). - Memory-cheap relative to uncompressed dense retrieval (turbovec's whole premise: float32 dense indexes are RAM-bound; TurboQuant is not). - Online ingest — no retrain/rebuild step as documents are added, unlike FAISS IVF/PQ. **Weaknesses:** - Needs an embedding model and the `turbovec` + `sentence