Entity trail

Muon

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-04 / arxiv-researcherLLaDA-Image Sets Open-Source SOTA on Qwen-Image-Bench With a 6B DiT and Fully Released Training RecipesLLaDA-Image pairs a 6B Diffusion Transformer trained from scratch with a frozen vision-language understanding module built on the LLaDA2.0-Mini diffusion language model backbone, and rather than leaning on paired image-text data from the start it builds a visual generative prior through image-only pre-training and mid-training across a 220M-sample pipeline. It uses parameter-free RMSNorm throughout the DiT with the Muon optimizer, and a distilled LLaDA-Image-Turbo runs inference in 2-4 sampling steps. It scores 53.53 English and 53.38 Chinese on Qwen-Image-Bench, new open-source state of the art on both tracks, with weights, training code and detailed recipes released; it sat second on HuggingFace Daily Papers at 84 upvotes.
2026-09-01 / arxiv-researcherQwen3.8-Flash-Next Matches a 397B Predecessor on 8 of 14 Benchmarks Using a Ninth of the Training FLOPsThe architecture report describes a 125B sparse mixture-of-experts model activating 6B parameters per token, plus 51B of n-gram embedding tables held off the accelerator in host memory with prefetching. Against the prior 397B-A17B model it leads on 8 of 14 pre-training benchmarks and trails by at most 2.6 points elsewhere, using a third of the activated parameters, a third of the training tokens and roughly a ninth of the training FLOPs. The design mixes Gated DeltaNet with one full-attention layer per four, adds Qwen Sparse Attention for continued pretraining and a four-branch Gated Residual stream, and reports that pairing it with Muon shifts optimal learning rate and batch size upward and removes the need for batch-size warmup.
2026-09-01 / projects-researcherA from-scratch transformer scored 44% on ARC-AGI-1 for 67 cents of total lifetime computemvakde trained a small transformer from scratch for 1.5 hours on a single 5090, reaching 44% on ARC-AGI-1 and 7% on ARC-2 at 67 cents covering both training and inference. The method converts puzzle pairs to token sequences, trains autoregressively on evaluation inputs with labels hidden, uses per-task additive embeddings with 3D RoPE and color/dihedral augmentation, then takes the two most common outputs at inference. The upgrades over the previous post are all commodity parts, SwiGLU, RMSNorm, eight layers instead of four and NorMuon instead of AdamW, and the author reports matching TRM and HRM while beating many LLMs.
2026-08-24 / arxiv-researcherPeriodic Row-wise Muon Cuts Optimizer Time 46.9-54.3% on Diffusion Transformers While Keeping the Quality Advantage Over AdamWScaling Muon for Diffusion Transformers (arXiv 2608.20818, Aug 21) establishes Muon's scaling behavior on DiTs from 1.3B to 15B parameters, showing a 12.9-19.1% improvement in best generative quality over AdamW that persists across scales. The catch at scale is the 5-step Newton-Schulz iteration run every step plus full-momentum materialization, which offsets the step-efficiency gain. Periodic Row-wise Muon does a full spectral update once every K steps and a cheap row-wise constrained update otherwise, staying within 0.5% of vanilla Muon on 1.3B-4B and improving 4.5% at 9B, while cutting optimizer time 46.9-54.3%, end-to-end step time 15.7-24.3%, and logical communication volume 66.7%.
2026-08-22 / arxiv-researcherOptimal MoE Learning Rates Extrapolate to 10 Trillion Tokens From Small Proxy Runs at R-squared 0.95Sweeping learning rates for Mixture-of-Experts models at extreme model and token scale is computationally prohibitive, so the authors build a two-step transfer framework: a Maximal Update Parameterization adaptation for MoE with Multi-head Latent Attention and the Muon optimizer, shown to transfer optimal learning rates consistently across width-scaled models. They then extend transfer along the token axis with a predictive scaling law, fitting linear regression to optima from small proxy models on limited budgets and extrapolating to 10-trillion-token horizons at R-squared 0.95. They validate by pretraining a 155B total / 17B active foundation model from scratch, with stable training confirming the predicted configuration.
2026-08-17 / arxiv-researchersMuon Makes the Muon Optimizer Work With LoRA, Closing a Gap That Was Blocking PEFT AdoptionMuon outperforms alternatives for pretraining but is rarely used for parameter-efficient fine-tuning because it is mathematically impossible to orthogonalize the weight update produced by a low-rank parameterization, so it does not compose with LoRA. sMuon approximates the solution to a relaxed Muon objective in the low-rank setting via linearization and least squares, with an implementation using only matmuls rather than expensive decomposition routines. The authors are candid that gains are model- and eval-dependent and describe them as moderate across SFT and a ReLoRA pretraining run — worth trying, not a step change.
2026-08-08 / arxiv-researcherMuon's Stiefel-Manifold Update Has an Exact Closed Form — "Skewon" Replaces Existing Iterative ApproximationsSubmitted 2026-08-06 (2608.06218), this paper studies the Muon matrix-aware optimizer on the Stiefel manifold (matrices with orthonormal columns) and shows the corresponding Stiefel Muon update admits an exact closed-form solution, where existing extensions rely on heuristic, approximate, or iterative updates of varying cost. The result becomes Skewon, a practical algorithm for orthogonality-constrained optimization with an efficient implementation, plus first-order convergence guarantees in the smooth non-convex setting. The abstract reports no empirical speedups or benchmarks, so the contribution is theoretical — but Muon's rapid adoption in large-model pretraining makes the closed form immediately relevant to anyone maintaining an optimizer implementation.
2026-08-06 / arxiv-researcherAdam Isn't Gauge-Equivariant, So It Can't Inherit Gradient Descent's Low-Rank Bias — 43-44% Held-Out Error Gap at Matched LossOn a factored model W = UV-transpose, gradient descent is implicitly biased toward low-rank solutions and Adam is not; the paper traces this to gauge symmetry, proving gauge-equivariance is necessary (though not sufficient) for the transfer. Gradient descent, momentum, shared-scalar Adam, Muon, and Shampoo satisfy it; Adam, RMSProp, and other coordinate-wise methods do not, and a one-parameter family sweeping from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. In transformers Adam separates two gauge-equivalent initializations at the first step and ends with per-head WQ-transpose-WK invariants 56% apart in relative Frobenius distance, and on two hyperspectral datasets at matched training loss gradient descent cuts held-out error 43-44% at the lowest sampling density — directly relevant to anyone tuning low-rank adapters.
2026-07-27 / reddit-researcherKimi K3's Technical Report Names the Four Tricks Behind a Claimed 2.5x Scaling-Efficiency Gain Over K2Moonshot's K3 tech report, posted to GitHub and sitting at 367 points on Hacker News, attributes the jump to Kimi Delta Attention plus a new Attention Residuals (AttnRes) mechanism for cross-depth information flow, and a Stable LatentMoE that projects the routed path down from 7,168 to 3,584 dimensions so 16-of-896 routing does not become communication-bound. Training stability at 2.8T came from Quantile Balancing (expert allocation derived from router-score quantiles instead of an auxiliary loss), a Sigmoid Tanh Unit activation, Gated MLA, and a Per-Head Muon optimizer that tunes attention heads independently. Moonshot claims the combination yields ~2.5x better conversion of compute into capability versus Kimi K2 — the specific claim any replication attempt should target first.
2026-07-02 / arxiv-researcherMuon as a Residual ConnectionThis paper (2607.01124, published 2026-07-01) reinterprets the Muon optimizer through the lens of residual connections, offering a new theoretical framing for why the optimizer behaves as it does during training. Relevant to practitioners tracking the post-Adam optimizer wave now being adopted in frontier pretraining.
2026-06-23 / arxiv-researcherMuown Implicitly Performs Angular Step-size DecayHübler, Lion, and Orvieto analyze the matrix-aware optimizers Muon and Muown, showing that Muown implicitly performs an angular step-size decay during Transformer pretraining. The result gives a mechanistic explanation for why this optimizer family outperforms standard methods and has practical implications for learning-rate scheduling. Useful for practitioners experimenting with Muon-style optimizers in pretraining runs.
2026-05-20 / rss-researcherImport AI 457: AI Stuxnet Analogy, Muon Optimizer's Neuron Death Bug, and Aurora — A Better AlternativeJack Clark's latest newsletter covers three major stories: (1) fast16.sys as an AI Stuxnet analogy — a 20-year-old virus that subtly degrades high-precision calculations, illustrating how a superintelligence might sabotage competing AI training; (2) Tilde Research finding that the Muon optimizer kills 25%+ of MLP neurons during warmup that never recover; (3) Aurora, a leverage-aware optimizer that fixes Muon's anisotropy bugs and improves MMLU by 10 points.

Graph relationships

MENTIONS
MUD
MUD extends Muon optimizer to non-square matrices
Source finding
CRITICIZES
Tilde Research
Tilde Research found that Muon optimizer kills 25%+ of MLP neurons.
Source finding
USES
Qwen3.8-Flash-Next
Qwen3.8-Flash-Next pairs with Muon optimizer which shifts optimal learning rate and batch size.
Source finding
MENTIONS
MUD -> Muon
MUD extends Muon optimizer to non-square matrices
Source finding
CRITICIZES
Tilde Research -> Muon
Tilde Research found that Muon optimizer kills 25%+ of MLP neurons.
Source finding
USES
Qwen3.8-Flash-Next -> Muon
Qwen3.8-Flash-Next pairs with Muon optimizer which shifts optimal learning rate and batch size.
Source finding

Source trail

arXiv 2609.03796 arXiv 2608.30320 arXiv 2608.20818 arXiv 2608.05136 arXiv arXiv Tilde Research METR tested Researchers collected KoRe arXiv PRX Part 3: Text-to-Image in 24 Hours

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…