Distill instead of fine-tune when the goal is inference cost, not new behavior.

LoRA/QLoRA adapts behavior but leaves a same-size, same-cost model. If you want cheaper inference, distillation gives you a permanently smaller one. A distilled Llama 3.1 8B trained on a 70B teacher captures 90 to 95% of quality at ~10% of inference cost. The hybrid move: distill for size, then rank-stabilized LoRA on the small model for domain fit. Source: DEV Community