algo-nlp-nerlisted

# Named Entity Recognition ## Overview NER identifies and classifies named entities in text into predefined categories (Person, Organization, Location, Date, Money, etc.). Approaches: rule-based (regex, gazetteers), statistical (CRF), neural (BiLSTM-CRF, transformer-based). Modern NER uses spaCy or Hugging Face models with F1 scores 85-95%. ## When to Use **Trigger conditions:** - Extracting structured entities from unstructured text - Building knowledge graphs from documents - Preprocessing for information retrieval or question answering **When NOT to use:** - For text classification (categorizing whole documents, not extracting entities) - For relation extraction between entities (need additional RE model) ## Algorithm ``` IRON LAW: NER Performance Depends on DOMAIN Match A model trained on news text (OntoNotes) performs poorly on medical records or legal documents. Domain-specific entities (drug names, legal citations, product SKUs) require domain-specific training data or fine-tuning. Always evaluate on YOUR domain's data. ``` ### Phase 1: Input Validation Determine: target entity types (standard: PER, ORG, LOC, DATE, MONEY or custom), input language, domain. Select appropriate pre-trained model or prepare training data. **Gate:** Entity types defined, model or training data available. ### Phase 2: Core Algorithm **Pre-trained model approach:** 1. Load model (spaCy, Hugging Face NER pipeline) 2. Process text through the pipeline 3. Extract entity spans with type