Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

In 30 Sekunden

Was passiert ist

Researchers found that standard knowledge distillation during mid-training hurts factual recall in smaller language models, and propose Switch Distillation to fix this.

Warum es zählt

Matters for engineers training smaller models via distillation from larger teachers, especially during intermediate training phases on curated data.

Achtung

Results are from controlled experiments; real-world effectiveness across diverse training setups and model scales remains to be validated in practice.