lit-deduplisted

# Lit-Dedup (Stage 5 — Merge & Deduplicate) Merge multiple pipeline stage outputs into one deduplicated master list with provenance tracking. **Input**: one or more JSON files from Stages 1–4 **Output**: `merged_results.json` + `merged_results.ris` + `dedup_log.json` ## Quick Start ```bash # Merge all JSON files in a directory python scripts/lit_dedup.py --input-dir ./results/ -o merged_results.json # Merge specific files python scripts/lit_dedup.py --inputs stage1.json stage2.json stage4.json -o merged.json # DOI-only dedup (skip LLM pass) python scripts/lit_dedup.py --inputs *.json --no-llm -o merged.json # Non-interactive (skip confirmation) python scripts/lit_dedup.py --inputs *.json -o merged.json --yes ``` ## Prerequisites ```bash pip install aiohttp ``` ## API Keys - `DEEPSEEK_API_KEY` — primary (DeepSeek chat API for fuzzy matching) - `ANTHROPIC_API_KEY` — fallback (Claude Sonnet via Anthropic API) ## CLI Flags | Flag | Default | Description | |------|---------|-------------| | `--inputs` | — | Input JSON files (nargs='+') | | `--input-dir` | — | Directory to glob *.json from | | `-o, --output` | `merged_results.json` | Output JSON path | | `--model` | `deepseek-chat` | LLM model override | | `--concurrency` | `100` | Max simultaneous LLM requests | | `--no-llm` | off | Skip Pass 2 (DOI-only dedup) | | `--yes` | off | Skip HITL confirmation checkpoint | | `--emit-pairs PATH` | — | Agent-driven: write candidate pairs to PATH and stop before the LLM pass (n