Rethinking On-Policy Distillation of Large Language Models II: One Training Example

In 30 Sekunden

Was passiert ist

Researchers show on-policy distillation of language models improves significantly when trained on just one query, recovering most gains from full-dataset training.

Warum es zählt

Matters for engineers optimizing LLM training efficiency and post-training workflows where data and compute budgets are constrained.

Achtung

The student model still requires hundreds of training steps to absorb supervision, suggesting the bottleneck is learning speed, not data quantity.