grpo-rl-training

# GRPO/RL Training with TRL Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions. ## When to Use This Skill Use GRPO training when you need to: - **Enforce specific output formats** (e.g., XML tags, JSON, structured reasoning) - **Teach verifiable tasks** with objective correctness metrics (math, coding, fact-checking) - **Improve reasoning capabilities** by rewarding chain-of-thought patterns - **Align models to domain-specific behaviors** without labeled preference data - **Optimize for multiple objectives** simultaneously (format + correctness + style) **Do NOT use GRPO for:** - Simple supervised fine-tuning tasks (use SFT instead) - Tasks without clear reward signals - When you already have high-quality preference pairs (use DPO/PPO instead) --- ## Core Concepts ### 1. GRPO Algorithm Fundamentals **Key Mechanism:** - Generates **multiple completions** for each prompt (group size: 4-16) - Compares completions within each group using reward functions - Updates policy to favor higher-rewarded responses relative to the group **Critical Difference from PPO:** - No separate reward model needed - More sample-efficient (learns from within-group comparisons) - Simpler to implement and debug **Mathematical Intuition:** ``` For each p...