Preference tuning and RL
After SFT, preference methods teach a model which of two outputs is better; RL with verifiable rewards teaches it to solve checkable tasks.
Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.
Key ideas
- DPO: trains directly on chosen/rejected pairs; no reward model needed.
- RLHF with PPO: reward model plus RL; powerful but complex.
- GRPO and related RLVR methods: sample several answers, reward the ones that pass a checker.
- Reinforcement fine-tuning services let you supply a grader instead of labelled answers.