GRPO, popularised by DeepSeek-R1 for training reasoning models, drops PPO's learned value network. For each prompt it samples a group of G completions, scores them (e.g. 1 if the final answer is correct), and uses each completion's reward relative to its group as the advantage:
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial