fine-tuning-with-trl

# TRL - Transformer Reinforcement Learning ## Quick start TRL provides post-training methods for aligning language models with human preferences. **Installation**: ```bash pip install trl transformers datasets peft accelerate ``` **Supervised Fine-Tuning** (instruction tuning): ```python from trl import SFTTrainer trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, # Prompt-completion pairs ) trainer.train() ``` **DPO** (align with preferences): ```python from trl import DPOTrainer, DPOConfig config = DPOConfig(output_dir="model-dpo", beta=0.1) trainer = DPOTrainer( model=model, args=config, train_dataset=preference_dataset, # chosen/rejected pairs processing_class=tokenizer ) trainer.train() ``` ## Common workflows ### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO) Complete pipeline from base model to human-aligned model. Copy this checklist: ``` RLHF Training: - [ ] Step 1: Supervised fine-tuning (SFT) - [ ] Step 2: Train reward model - [ ] Step 3: PPO reinforcement learning - [ ] Step 4: Evaluate aligned model ``` **Step 1: Supervised fine-tuning** Train base model on instruction-following data: ```python from transformers import AutoModelForCausalLM, AutoTokenizer from trl import SFTTrainer, SFTConfig from datasets import load_dataset # Load model model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B") tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B") # Load instruction datase...