Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-24 · high
One model scored 99.3 on safety but refused a third of harmless prompts, and a distillation shortcut crashed another's robustness score to 2.6.
Why now: The paper posted in August 2026, covering 120 open-weight models in one benchmark.
A benchmark called aiXamine ran more than 5,000 tests on 120 language models and found top safety scores carry a hidden cost. One model scored 99.3 on safety alignment, then refused one in three benign, harmless queries, per the aiXamine paper.
That tradeoff matters for anyone picking an open-weight model off a leaderboard. A high safety score can mean careful tuning, or it can mean a model trained to refuse by default, and the ranking doesn't show which.
The paper's second finding is sharper. Testing the same base architecture under different training methods, aiXamine found that off-policy distillation without on-policy correction dropped robustness from 56.9 to 2.6.
A drop that size, on the same base architecture, wouldn't show up in a single safety number.
Anyone deploying an open-weight model needs the refusal rate on benign traffic and the training method behind the safety number. The paper doesn't say how many of the 120 models it tested shipped with that gap unmeasured.
Each link below shares sources, entities, or timing with this story.
RAGAS-style evaluation checks correctness against a frozen snapshot, which means routine document updates and corrections can silently break production without moving a dashboard. This ASE 2026 paper defines 11 mutation operators perturbing at both the pre-chunk index level an...
Adversaries inject documents into RAG knowledge bases that trigger safety refusals on benign queries. Weaponizes alignment homogeneity itself — high cross-model transfer rates. The alignment-as-vulnerability paradox. (arXiv 2603.03919)
13 public sources consolidated into 9,740 skills (7,505 malicious, 2,235 benign) across 11 harmonized attack categories. Learned text detectors score 0.882-0.932 Macro-F1 under random splits but collapse to 0.653-0.665 source-disjoint. arXiv Three off-the-shelf skill scanners...
A 1.5B distilled model trained with GRPO chooses NoThink, Short, or Long at response start, using a shaped reward that makes each mode pay off at a different length plus hard per-mode token caps. Accuracy held at 0.782 against 0.796 baseline while mean length fell from 4,796 t...
Three rounds of LoRA self-training on Qwen3-8B against a frozen control turned up seven systematic measurement failures, including a ledger showing capability changes on a model that was never trained, largely an artifact of inference batching. arXiv After a per-problem exact...
Multiple independent models train against each other with peer-derived rewards and no ground-truth labels, gaining 3.0-8.6% across seven text benchmarks and 2.3-7.2% across four multimodal ones. The mechanism claim matters more than the numbers: varying architectures, model si...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.