Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-06-30 · high
The decision rule: RAG for knowledge that changes, fine-tuning for stable behavior, format, and tone failures.
Why now: bigdataboutique.com is pitching this order as the 2026 consensus for the RAG-versus-fine-tuning call, not just one team's preference.
Skip fine-tuning until prompting and retrieval fail, per bigdataboutique.com's rundown of the 2026 tuning order. Reach for a full fine-tune before trying those two and you're burning compute and engineering time on something a cheaper prompt might have fixed.
The full sequence runs prompt, then RAG, then a thin fine-tune, then distillation, exhausting each cheaper option before the next.
Even the fine-tune step stays thin. The consensus favors a LoRA or QLoRA adapter paired with retrieval over a full fine-tune, per bigdataboutique.com.
On preference data, DPO is now the default over RLHF whenever preference pairs already exist. Distillation is last, reserved for teams that need a smaller model to ship.
Once preference pairs exist, DPO is the default and running RLHF from scratch becomes the choice that needs justifying, not the reverse.
bigdataboutique.com is pitching this order as the 2026 consensus for the RAG-versus-fine-tuning call, not just one team's preference.
Each link below shares sources, entities, or timing with this story.
PortLLM claimed training-free, data-free transfer of LoRA patches onto updated base models, but only over short horizons and without theoretical grounding. This study runs 10 continual-pretraining steps on Mistral, Gemma, and Qwen and finds portability persists long-run, meani...
Nathan Lambert doesn't hand out "step change" lightly, so when his June 22 Interconnects essay called GLM-5.2 "the step change for open agents," I read it twice. His argument is sharper than the usual "strong open model" take. Static intelligence benchmarks stopped mattering m...
LoRA/QLoRA adapts behavior but leaves a same-size, same-cost model. If you want cheaper inference, distillation gives you a permanently smaller one. A distilled Llama 3.1 8B trained on a 70B teacher captures 90 to 95% of quality at ~10% of inference cost. The hybrid move: dist...
Unsloth updated all uploads with corrected chat templates, explicit <|think|> thinking control, and fixes for grad accumulation exploding losses (300-400 down to 10-15). Old versions have broken chat behavior and a 26B/31B inference IndexError.
New batching algorithms enable ~7x, up to 12x+, longer-context GRPO training with no accuracy or speed penalty versus optimized FA3 and chunked-loss setups (Unsloth Docs). Qwen3-8B GRPO reaches 110K context on one 80GB H100 via vLLM plus QLoRA. For solo builders doing reasonin...
Researchers reveal that Direct Preference Optimization implicitly operates over a full preference graph, meaning it extracts more signal from existing datasets than anyone realized. Practical implication: your existing RLHF data may be more valuable than you think.
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.