The Correctness Illusion in LLM-Generated GPU Kernels

arXiv·medium signal

Argues current GPU-kernel benchmarks use weak tests that miss real bugs. Using op-schema-aware seeded fuzzing against a high-precision fp64 CPU reference on 24 Triton kernels (15 correct, 9 intentionally buggy), the method caught all 9 buggy variants and passed all 15 controls across five GPU classes — errors standard benchmarks would have missed. A caution for anyone trusting LLM-generated kernels or kernel-generation leaderboards.