PPO, used in classic RLHF, improves a policy while preventing any single update from moving it too far. For each token or action, with log-probs under the new and old policies and an advantage estimate :
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial