Distillation and synthetic data

Use a strong model to generate or label data, then train a smaller, cheaper model to match it on your task.

Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.

Key ideas

• Generate inputs from real distributions, label with the teacher, filter with evals.
• Diversity matters more than volume; deduplicate and cover edge cases.
• Check provider terms: some forbid using outputs to train competing models.

Common mistakes

• Training on unfiltered synthetic data, which bakes in the teacher's mistakes.