text-classificationlisted

Designs and validates LLM-based text classification for research data — codebook construction, choice of learning regime, model selection and reproducibility, prompt construction, pilot validation against human coding with agreement statistics (kappa, F1), hybrid human-LLM workflows, and reporting model-coded data. Also carries a resumable batch pipeline with a rule-based baseline for coding large sets of repeated free-text values against a closed codebook — occupation, institution, and registry text, and open-text survey responses — with a frozen input set, incremental output, residual buckets, stratified hand validation, a regex or dictionary comparison, and a published lookup table. Use when the user asks to classify, code, or label text at scale. Discovering categories rather than applying them goes to topic-modeling.
scdenney/open-science-skills · ★ 57 · AI & Automation · score 74

Install: claude install-skill scdenney/open-science-skills

# LLM-Based Text Classification for Social Science Research ## Instructions ### 1. Codebook Design - Before drafting the codebook, specify the population, sampling frame, and (for experimental data) the treatment condition each response is drawn from. These constrain which categories can plausibly exist and which demographic subgroups any bias assessment must cover. LLM classification extends, rather than replaces, the longer open-ended coding tradition in survey methodology (Geer 1988; Lupia 2018). - Treat codebook design as the most consequential decision in the classification pipeline. LLMs struggle with loose instructions and revert to general-purpose definitions rather than following researcher-specific operationalizations (Halterman & Keith 2025). - Structure each code with the following components (adapted from Halterman & Keith 2025): - **Label**: The exact output string the model should return - **Definition**: A single-sentence operationalization of the construct - **Clarification**: What IS included — boundary cases that belong in this category - **Negative clarification**: What is NOT included — common confusions and adjacent categories - **Examples**: 2-3 positive examples (correctly classified) and 2-3 negative examples (common misclassifications) - Keep the number of codes small (3-6) for initial classification. Larger coding schemes increase ambiguity and reduce inter-annotator agreement for both humans and LLMs (Chae & Davidson 2025). - Allow mul