Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
The architecture funnels local features into proxy states per layer, running attention there instead of over every token.
Why now: The trade-off numbers come from a single arXiv posting, with no comparison yet against other long-context compression methods on the same hardware.
ProxyFormer trains sequences of about 700,000 tokens on a single 16GB GPU, where a standard transformer decoder manages about 20,000 tokens on the same card, according to the ProxyFormer paper. For anyone training long-context models on one consumer or workstation GPU, this closes the gap between a short document and a full codebase or long transcript.
The architecture squeezes local features into a small set of proxy states per layer, working bottom-up. Global attention runs across that compressed set, not over every token directly. The results decompress and feed back into a local stream that persists across layers, so a detail one compression pass misses stays recoverable later.
The full trade-off happens at a compression ratio of 64, batch size 1. A model trained with a 64,000-token window held 92 to 95% accuracy on a multi-needle retrieval test, and the design adds a proxy-only key-value cache scheme for inference. The paper doesn't say whether pushing the compression ratio past 64 keeps that accuracy intact.
Each link below shares sources, entities, or timing with this story.
arXiv 2608.08097 exploits decode-time attention sparsity: keep a 2,048-token budget resident and speculatively prefetch the rest via lookahead prediction. Reported results are 1.69x speedup on reasoning workloads, up to 2.1x on multi-GPU long-context serving, roughly 2x throug...
arXiv 2608.04074 reframes KV quantization as transform coding where distortion is measured on the attention product, deriving closed-form optimal transforms from calibration statistics that satisfy a generalized Parseval relation. At two bits per element it recovers most of th...
Hardware-aware compression that preserves bit-exact model outputs while reducing memory footprint and increasing throughput. Unlike quantization, zero accuracy tradeoff — drop-in safe for production inference pipelines. Substantial memory reduction on standard GPU stacks with...
Load Hijack modifies nothing but router weights in a checkpoint. When a private trigger appears, token-to-expert assignment concentrates on experts co-located on a single GPU, making it a straggler while peers idle (arXiv 2608.10614). Across three MoE families and four corpora...
Using op-schema-aware seeded fuzzing against a high-precision fp64 CPU reference on 24 Triton kernels, 15 correct and 9 intentionally buggy, the method caught all 9 buggy variants and passed all 15 controls across five GPU classes (arXiv:2606.20128). Standard kernel benchmarks...
SOL-ExecBench measures AI-generated GPU kernels against theoretical hardware speed-of-light limits rather than relative rankings. Current agentic systems achieve 40–70% of theoretical hardware efficiency, with clear headroom. As agents increasingly generate and optimize GPU co...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.