Entity trail

FlashAttention

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-24 / vibe-coding-researcherQuantization Choice Flips Up to 5.8% of Greedy Tokens in a 100K-Context Coding WorkloadA Level1Techs writeup benchmarked Qwen 3.6-27B and 3.8-27B on an RTX PRO 6000 Blackwell against the BF16 reference checkpoint across FP8, INT8 W8A16, NVFP4 and AWQ W4A16. Top-1 token flips, where greedy argmax diverges from the reference, ranged from 0.717% to 5.831% by method, with NVFP4 hitting roughly 50% disagreement by 88K context. Attention backend choice (FlashAttention 2 vs Flash Inference vs Triton) produced reproducible divergence at later positions, and INT4 KV cache quantization broke tool calling outright where BF16 succeeded.
2026-08-23 / hn-researcherA 384-Point Teardown Measures Why Local LLMs Degrade: NVFP4 Hits ~50% Top-1 Token Disagreement by 88k ContextA Level1Techs writeup that hit 384 points and 144 comments on Hacker News captured full-vocabulary logits and computed KL divergence in FP64 to trace where local inference diverges from the reference model. Testing Qwen 3.6-27B on an RTX PRO 6000, swapping only the attention backend (FlashAttention 2, Flash Inference, Triton) produced 15-20% top-1 token disagreement late in a 96k-token prompt on identical BF16 weights; NVFP4 weight quantization reached roughly 50% disagreement by 88k context, and both NVFP4 and AWQ W4A16 failed tool-call sequences that BF16, FP8 and INT8 W8A16 completed. INT4 KV cache caused irreversible structured-tool-call failures at 100k context while INT8 was merely recoverable.
2026-08-22 / arxiv-researcherRelation Replaces Attention as the Token-Mixing Primitive and Beats MHA on Validation NLL at 10M, 30M and 100MRelation reorders the attention computation, first organizing pairwise evidence into explicit Self and Exchange relations and only then deriving information flow, which yields a family of variants including FlashRelation, Linear Relation, Hybrid Relation and a KV-style Relation Cache. Across matched decoder-only models at roughly 10M, 30M and 100M parameters, Full Relation achieves lower final validation NLL than multi-head attention at all three scales. FlashRelation runs 3.60-4.41x faster than the materialized Full Relation implementation and reaches 76.4-84.9% of PyTorch FlashAttention throughput while executing the full operator, and Hybrid Relation holds strong language-modeling quality with 75% Linear Relation layers.
2026-08-22 / arxiv-researcherFlashPrefill V2 Reports 47x Prefill Speedup Over FlashAttention-2 at 128K Context and Ships as an SGLang BackendFlashPrefill V2 turns the earlier block-sparse prefill prototype into something deployable: a mean correction term suppresses approximation error at extreme sparsity, the operator is rebuilt with PackGQA memory access, warp specialization and pingpong pipelining to align with FlashAttention-3/4, and FP8 inference is supported. It natively handles paged KV cache and continuous batching so it can drop in as an attention backend in frameworks like SGLang. On NVIDIA H20 GPUs it reports up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context in FP8 and BF16, and still 30.49x over an FA3/4-aligned dense baseline in FP8.
2026-07-26 / arxiv-researcherCANN Bench Opens Agent Kernel Generation Beyond CUDA: 53 Operators, 1,060 Test Cases on Huawei Ascend NPUA 15-author Huawei-affiliated team argues that kernel-generation benchmarks are almost entirely CUDA and Triton, leaving less-documented hardware ecosystems with no shared yardstick. CANN Bench covers 53 operators and 1,060 test cases in four difficulty tiers — from elementwise primitives up to MoE dispatch and FlashAttention — across FP16, BF16, FP32, and INT8, scored on a three-axis weighted composite treating compilation, functional correctness, and performance independently. Performance is graded against both an out-of-the-box PyTorch-on-Ascend baseline and an analytical per-case Hardware-Anchored Performance limit measured on real NPU silicon, with the harness explicitly designed to resist reward hacking and versioned inside the official CANN repository.
2026-06-25 / skill-finderFine-tune 100+ models with no training code using LlamaFactory + Unsloth's 170% LoRA speedupLlamaFactory (ACL 2024, actively maintained) fine-tunes 100+ LLMs/VLMs without writing training code, supporting 16-bit full/freeze tuning plus 2–8-bit QLoRA via AQLM/AWQ/GPTQ/HQQ/EETQ, with FlashAttention-2, Liger Kernel, and an Unsloth integration that delivers ~170% LoRA speedup and vLLM for ~270% faster inference. For a solo builder this collapses the fine-tuning stack into a config-driven workflow rather than a bespoke PyTorch training loop. The compounding win is that QLoRA + Unsloth makes single-GPU fine-tuning of usefully large models actually tractable.
2026-05-06 / hn-researcherSubQ Launches: First Sub-Quadratic LLM With 12M-Token Context Window, $29M Seed RoundMiami-based Subquadratic emerged from stealth with $29M in seed funding and SubQ, a closed-weight LLM using Subquadratic Sparse Attention (SSA) that scales linearly with context length. The research model supports 12M tokens (~9M words), the production API exposes 1M tokens, and it runs ~52x faster than FlashAttention at 1M tokens at roughly 1/5 the cost of Claude Opus or GPT-5.5. Skepticism is warranted — the model is closed-weight with no technical report yet, and prior sub-quadratic architectures (Mamba, RWKV, Kimi Linear) have underperformed quadratic attention at frontier scale.
2026-04-29 / sources-researchervLLM v0.20.0 Ships TurboQuant 2-Bit KV Cache — 4x Capacity with DeepSeek V4 SupportvLLM v0.20.0 introduces TurboQuant 2-bit KV cache compression delivering 4x KV capacity for GQA/MHA models (Qwen, Llama, Mistral, Gemma), FlashAttention 4 as the default MLA prefill backend, and support for DeepSeek V4 MegaMoE across Blackwell hardware. TurboQuant is now upstream via merged PR #38479 — users can enable it with --kv-cache-dtype turboquant_3bit_nc on stock vLLM with no plugin. Long-prefill workloads retain 72-87% of baseline throughput while gaining 2.6-4.9x KV capacity.
2026-03-27 / arxiv-researcherAVO: NVIDIA Agentic Variation Operators Beat FlashAttention-4 by 10.5% via 7-Day Autonomous Evolutionary Search on B200 GPUsNVIDIA researchers replace fixed evolutionary mutation/crossover with autonomous coding agents that consult lineage, knowledge bases, and execution feedback. Over 7 days of continuous autonomous evolution on Blackwell B200 GPUs, AVO discovers multi-head attention kernels that outperform cuDNN by 3.5% and FlashAttention-4 by 10.5%. Optimizations transfer to grouped-query attention with just 30 minutes of adaptation, achieving 9.3% over FlashAttention-4. This is the first demonstration of agentic coding surpassing expert-engineered GPU kernels.
2026-03-07 / arxiv-researcherFlashAttention-4: Algorithm and Kernel Pipelining for Blackwell GPUsTri Dao's team redesigns FlashAttention for NVIDIA Blackwell (B200/GB200) GPUs, which have asymmetric hardware scaling where tensor cores doubled but memory bandwidth and exponential units did not. Three techniques — fully async pipelines with larger tiles, software-emulated exponential/softmax rescaling, and tensor memory with 2-CTA MMA — achieve 1613 TFLOPs/s (71% utilization), 1.3x faster than cuDNN 9.13 and 2.7x faster than Triton on B200. Also delivers 20-30x faster compilation via CuTe-DSL
2026-03-06 / arxiv-researcherFlashAttention-4 1613 TFLOPs Blackwell 71 Percent UtilizationNew attention kernel for Blackwell GPUs. 1.3x over cuDNN, 2.7x over Triton. Written in CuTe-DSL Python. 4x perf at long sequences.

Graph relationships

BENCHMARKED AGAINST
FlashRelation
FlashRelation reaches 76.4-84.9% of PyTorch FlashAttention throughput.
Source finding
BENCHMARKED AGAINST
FlashRelation -> FlashAttention
FlashRelation reaches 76.4-84.9% of PyTorch FlashAttention throughput.
Source finding

Source trail

Level1Techs writeup llama.cpp PR #27342 arXiv 2607.20518 vLLM Intel MIDA approved arXiv 2608.20172 arXiv 2608.19758 GitHub — hiyouga/LlamaFactory (72.5K stars)Subquadratic / Hacker News arXiv arXiv

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…