SMetric Rethinks LLM Scheduling for Agentic Serving, Where KV-Cache Reuse Tops 80%

arXiv·high signal

SMetric argues human-oriented LLM schedulers misfit agentic serving: agents only act on complete responses, so cluster tokens-per-second (TPS) becomes the real objective and per-token latency requirements relax rather than vanish. In a production trace from BAILIAN, agent requests share so much context that KV-cache reuse exceeds 80% of request tokens versus 54-62% for human traffic, motivating a balanced session-centric scheduler. Directly relevant to anyone self-hosting inference for agent fleets.