Mismatch Matters: On-Policy Distillation Beyond Token Agreement

In 30 Sekunden

Was passiert ist

Researchers identified a failure mode in on-policy distillation where student models achieve token agreement with teachers while producing globally flawed responses, and proposed TIDE to address token-level mismatches.

Warum es zählt

Engineers building LLM post-training pipelines should care when training smaller models to mimic larger ones, especially in mathematical reasoning tasks.

Achtung

TIDE was tested only on Qwen3 teacher-student pairs and mathematical reasoning benchmarks; generalization to other domains and model families remains unclear.