MIT/ETH Zurich SDFT: Self-Distillation Fine-Tuning Eliminates Catastrophic Forgetting in Domain Adaptation

VentureBeat·medium signal

Self-Distillation Fine-Tuning (SDFT) uses a frozen teacher version of the model prompted with expert demonstrations via ICL to supervise a student version that sees only the bare query, creating an internal feedback loop that teaches new skills without overwriting existing ones. On Science Q&A, SDFT hit 70.2% vs 66.2% for standard SFT, and crucially maintained all previously learned skills while adding new ones — something standard SFT and RL-based approaches both fail at. The tradeoff is approximately 4x slower training and 2.5x more FLOPs, making it best suited for high-stakes domain adaptation where capability regression is unacceptable.