transformers-config-tokenizers-expert

# Transformers config + tokenizers expert Target: engineers writing a preflight tool (or a vLLM/sglang operator) that must decide, before handing a HuggingFace snapshot to an inference engine, *which* files win, *which* tokens are structural, and *which* class will actually instantiate. Almost every major 2026 release has shipped with drift between `tokenizer_config.json`, `generation_config.json`, `config.json`, and the Rust-backed tokenizer state. The skill exists so a preflight tool can answer that drift authoritatively — not guess. --- ## Stance - **Cite, don't paraphrase.** Every load-bearing claim has a file:line or URL citation in `references/`. Point at the source. - **Version-gate.** Transformers v5 (GA 2026-01-26) renamed the tokenizer classes and changed serialization shapes. Pre-5.0 and post-5.0 diverge — check `transformers.__version__` before claiming. - **Rust is truth.** For any model with `tokenizer.json`, the authoritative added-token state is `tokenizer.backend_tokenizer.get_added_tokens_decoder()`. Python-side `all_special_ids` / `special_tokens_map` / `added_tokens_decoder` are views; treat them as such. - **Engines disagree.** vLLM and sglang both union-merge `generation_config.eos_token_id`, but apply it through different pipelines (see `engine-knobs.md`). Predict per engine, not in the abstract. --- ## Triage: symptom → layer → reference Use this table first. Deep dives live in `references/`. | Symptom | Layer | Open | |---...