AgentSysBench finds non-LLM components dominate latency in half of agents

AgentSysBench, a systems benchmark covering ten agentic applications with unified instrumentation, found non-LLM components dominate latency in 5 of the 10 apps tested. That contradicts the usual assumption behind agent infrastructure: the model call is the expensive part.

That matters because it changes what's worth optimizing first. Teams tuning inference servers while a CPU-bound sandbox eats the clock are working on the wrong layer.

Sandbox working sets peak at 28 GB per session. Latency across GPU-bound inference, memory-bound retrieval, and CPU-bound sandboxes diverges by up to 32x within the same app.

Production sessions also sit idle for minutes to hours between active steps, which breaks the always-hot assumption behind most LLM-serving designs.

What the fixes buy you

AgentSysBench tests four fixes against the same suite. Task-aware serving cuts latency 29 to 40%, and communication-aware placement gets up to 4.5x. Offloading state cuts memory 4.6x, and caching tool results removes 35.2% of redundant search calls.

The paper also names a control-plane tax, the auxiliary LLM calls and tool-schema context that crowd out the compute actually doing the work. It doesn't say whether the four fixes compose, so stacking all of them might not add up to the sum of their gains. The 32x latency spread inside a single app is the number worth checking first.