Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-03-17 · source-backed
Moonshot AI replaces standard fixed residual connections with softmax attention over preceding layer outputs. Already in production at 48B scale (Kimi Linear). Consistent scaling improvement validated across model sizes. 1,330 HuggingFace upvotes. Source
Each link below shares sources, entities, or timing with this story.
The minute Fable 5 and Mythos 5 went dark for foreign nationals, r/LocalLLaMA found its answer. Moonshot AI's Kimi K2.7 Code is a 1T-parameter MoE (32B active, 384 experts), 256K context, shipped under a Modified MIT license. The headline number that's getting it pulled: 81.1...
Moonshot AI (Kimi) published a paper proposing attention residuals to replace the residual connections used in every transformer since ResNet — routing skip-connection information through learned attention weights instead of direct addition. Results "look legit" at tested scal...
Cursor shipped Composer 2 on March 19, marketing it as a proprietary in-house model. Within 24 hours, a developer found the API routing to kimi-k2p5-rl-0317-s515-fast. The model powering the most-hyped coding tool update of the month was Moonshot AI's Kimi K2.5 with continued...
Sebastian Raschka's July 28 teardown argues K3 is less exotic than the release framing suggests: a scaled production version of Kimi Linear with Kimi Delta Attention as the hybrid attention layer and LatentMoE compressing large linear layers by down-projection. The genuinely n...
The file is 1.56TB. That's the first thing you notice about the moonshotai/Kimi-K3 Hugging Face repo that went live today. 2.8 trillion total parameters, 104B activated, 896 routed experts with 16 selected plus 2 shared per token, 93 layers split 69 Kimi Delta Attention and 24...
Moonshot AI released Kimi K3, a sparse mixture-of-experts activating 16 of 896 experts per token. That's about 1.8% of the pool live at any moment, with a 1M-token context window and native vision. Two new architectural pieces show up: Kimi Delta Attention and Attention Residu...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.