knowledge-distillation

# Knowledge Distillation: Compressing LLMs ## When to Use This Skill Use Knowledge Distillation when you need to: - **Compress models** from 70B → 7B while retaining 90%+ performance - **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral) - **Reduce inference costs** by deploying smaller student models - **Create specialized models** by distilling domain-specific knowledge - **Improve small models** using synthetic data from large teachers **Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation **Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116) ## Installation ```bash # Standard transformers pip install transformers datasets accelerate # For training pip install torch deepspeed wandb # Optional: MiniLLM implementation git clone https://github.com/microsoft/LMOps cd LMOps/minillm pip install -e . ``` ## Quick Start ### Basic Knowledge Distillation ```python import torch import torch.nn.functional as F from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments # 1. Load teacher (large) and student (small) models teacher = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-70b-hf", # Large teacher torch_dtype=torch.float16, device_map="auto" ) student = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", # Small student torch_dtype=torch.fl...