Showing the teacher the answer adds less to self-distillation than the distillation itself does
arXiv 2609.20612 (2026-09-17) isolates what privileged information actually contributes to on-policy self-distillation by building AMPLE-Math, a reusable suite of 5,319 math problems with six reasoning views that share the same answer, then matching each view against reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for most of Qwen3-1.7B's gain under thinking-enabled evaluation, in domain and on external benchmarks; evidence for an extra reference benefit is modest in Qwen and strongest for a polished solution, while complete traces add two points in SmolLM3-3B at step 50. The benefit is student-dependent, so the common practice of feeding the teacher the gold answer is worth ablating before you pay for it.
↳ Follow the thread