Holding Back Ready Agent Turns Instead of Releasing Them Eagerly Cuts P95 Workflow Latency up to 3.50x
Agentic workflows interleave model turns with tool calls, so completion time depends on when ready turns are released, not just inference speed. Most runtimes release each turn the moment it is ready, which under contention accumulates released-but-unfinished work that the workflow-level policy can no longer reorder. This method jointly decides which ready turn to release and how much released-unfinished work to allow, using a mean-CVaR objective over the tail risk of unfinished workflows plus online estimates of turn work and an adaptive budget keyed to queue pressure. On real agent execution traces from software engineering tasks across multiple LLMs, it matches eager release under light load and delivers up to 3.50x lower P95 flow time under contention.
↳ Follow the thread