RoofLang Lets an Optimizer Agent Search Inference Architectures Instead of Profiling One Stack, Finding 6.2-50.1% Gains on B300

arXiv 2609.12551·medium signalXBlueskyLinkedInCopy link

Existing AI-driven inference optimization is profiling-based, which confines the search to whatever the current software stack can already do and rules out finding a fundamentally different architecture. RoofLang is a DSL supplying a general workload representation, a verifiable mutation space, and an implementation-independent evaluator so the loop can run without a profiler. It reveals that DeepSeek V4-series models could reach 3.5-39.5x higher peak decode throughput than other representative models, largely from compact KV-cache designs enabling larger batches, and a persistent optimizer agent discovered architectures improving DeepSeek V4 Pro throughput and interactivity on NVIDIA B300 by 6.23-50.1%.