Evolving Infra:面向智能體工作負載的大規模自適應 LLM 服務
Evolving Infra: Agent-Adaptive LLM Serving at Scale
Evolving Infra 是一套面向智能體工作負載的自適應服務框架,在 GLM-5.2 和 Kimi K2.6 的編碼、數學及電腦操作任務測試中,使用最多 256 GB300 GPUs,平均端到端請求延遲和 P95 TTFT 相較 Dynamo 分別降低最多 4.1 倍和 14 倍。
Published on Nov 9
·
on Nov 12
Authors:
,
,
,
,
,
,
,
,
,
Abstract
Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limitations lead to load imbalance between prefill and decode nodes and eviction of KV caches that agents will reuse. We present Evolving Infra, an agent-adaptive serving framework that adapts compute allocation to agent workloads and KV cache management to agent context management at runtime. We decouple compute allocation from key--value (KV) cache management, allowing nodes to switch efficiently between prefill and decode. Building on this design, an online planner matches the PD split to workload composition, while an agent-aware distributed KV manager guides cache eviction and transfer with signals from agent context management. Across coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, Evolving Infra reduces mean end-to-end request latency and P95 TTFT by up to 4.1x and 14x over Dynamo, NVIDIA's production framework.
View arXiv page View PDF Project page GitHub 9 Add to collection
Get this paper in your agent:
hf papers read 2611.12304
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.12304 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.12304 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.12304 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co