algo-nlp-ldalisted

# LDA Topic Modeling ## Overview Latent Dirichlet Allocation models each document as a mixture of topics and each topic as a distribution over words. Discovers K latent topics from a corpus without supervision. Uses Gibbs sampling or variational inference. Complexity: O(N × K × iterations) where N = total word tokens. ## When to Use **Trigger conditions:** - Discovering latent themes in a large document collection - Organizing/categorizing documents by automatically discovered topics - Exploratory text analysis when categories are unknown **When NOT to use:** - When categories are known (use supervised classification) - For short texts (tweets, titles) — too few words per document for reliable topic assignment - When you need semantic understanding (use embeddings) ## Algorithm ``` IRON LAW: The Number of Topics K Must Be Chosen, Not Discovered LDA does NOT tell you how many topics exist. K is a hyperparameter. Too few topics: overly broad, mixed themes. Too many: fragmented, redundant topics. Use coherence score (C_v) to compare K values, but the final choice requires human judgment on topic interpretability. ``` ### Phase 1: Input Validation Preprocess: tokenize, remove stop words, apply lemmatization. Build document-term matrix. Filter: remove terms appearing in <5 or >50% of documents. **Gate:** Clean DTM, vocabulary size reasonable (1K-50K terms). ### Phase 2: Core Algorithm 1. Choose K (start with √(N/2), try range K=5,10,15,20,...) 2. Set hyperparameters: α =