algo-ecom-bm25listed

# BM25 Ranking Function ## Overview BM25 (Best Matching 25) is an improved TF-IDF ranking function that adds term frequency saturation and document length normalization. Score = Σ IDF(t) × (TF(t,d) × (k₁+1)) / (TF(t,d) + k₁ × (1 - b + b × |d|/avgdl)). Standard parameters: k₁=1.2, b=0.75. The backbone of most text search engines (Elasticsearch, Solr). ## When to Use **Trigger conditions:** - Building product search with text-based relevance ranking - Replacing basic TF-IDF with better document length normalization - Tuning search relevance in Elasticsearch/Solr **When NOT to use:** - When semantic similarity matters more than keyword matching (use embeddings) - For single-field exact matching (simpler methods suffice) ## Algorithm ``` IRON LAW: BM25 Has Two Critical Parameters — k₁ and b k₁ controls term frequency saturation: higher k₁ = more weight to repeated terms. k₁=0 ignores TF entirely (boolean). b controls document length normalization: b=1 fully normalizes by length, b=0 ignores length. Default k₁=1.2, b=0.75 works for most cases but MUST be tuned for your specific corpus. ``` ### Phase 1: Input Validation + Tokenization Tokenize each document to lowercase word tokens. **Remove stop words** before counting — the bundled script drops a standard English stop list (`the, a, an, and, or, but, of, in, on, at, to, for, with, by, from, as, is, are, was, were, be, been, being`). Then build an inverted index: term → list of (document, term frequency). Compute: document