llm-d 1.0 在 PyTorch Conference North America 發佈 Kubernetes 原生分散式推理服務堆疊

Contributors to the llm-d project announced llm-d 1.0, a Kubernetes-native distributed inference serving stack, on October 20, 2026, at the PyTorch Conference North America, according to an IBM Research announcement. IBM said the release moves llm-d beyond demonstrating individual performance techniques and is meant to help organizations bring model deployment in-house as inference costs rise.

llm-d is built to deploy inference stacks for any model, on any accelerator, in any cloud. Red Hat, IBM, and Google are the project’s lead maintainers and co-founded llm-d with CoreWeave and NVIDIA; AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI have joined as contributors. The Cloud Native Computing Foundation (CNCF) hosts llm-d as a sandbox project. IBM describes the result as a vendor-neutral alternative for organizations that want what it calls frontier-class inference performance without depending entirely on one proprietary model provider, cloud, or hardware platform.

Carlos Costa, a Distinguished Engineer at IBM Research and llm-d co-founder and lead maintainer, said version 1.0 adds the maturity to run llm-d like any other critical service, pointing to a stable API, multiple tenants with competing priorities sharing the same infrastructure, and upgrades that do not interrupt operations.

What Version 1.0 Adds

The project’s v1.0.0 release notes, published on GitHub on October 20, 2026, with signed and verified tags, lead with multi-tenant quality of service: priority- and fairness-aware flow control with controlled load shedding, including a probabilistic admitter, program-aware fairness, and turn-priority scoring, so latency-sensitive requests and bursty agent workloads can share a fleet without starving one another under overload. Flow-control configuration is consolidated under a single flowControl field.

For zero-downtime operations, the router gains high availability with hot standby and safe rolling updates, and a DisaggregatedSet controller manages prefill and decode replicas as one logical unit, with coordinated rollouts, lockstep upgrades, readiness gated on Pod snapshot, and native revision routing. The router configuration API graduates to v1, the Responses API supports previous_response_id, and the endpoint-picker request path is hardened for reliability, performance, and scalability. A topology-aware accelerator guide for DRA and DRANET is included.

Agent-aware serving adds a session-state-producer plugin and evolving session affinity to keep multi-turn conversations close to their cached context, with turn-priority scheduling and Responses API state to help agents reuse context across turns. The release aligns the project’s components at v1.0.0, adds ecosystem pieces including an integration with ai-dynamo/modelexpress for peer-to-peer loading of model weights, the llm-d-api-extensions package, and the llm-d-pd-utils diagnostics tooling, and deprecates the project’s own llm-d-rocm container image builds. Model-server images include vLLM and SGLang builds across accelerator targets, and the prefill/decode disaggregation guide now requires LeaderWorkerSet v0.11.1 or newer with the DisaggregatedSet API enabled.

Core Techniques and Flow Control

Since launch, the project has focused on five approaches to improving inference: prefix-cache-aware routing, which directs a request to infrastructure that may already hold its shared context; KV-cache offloading, which moves cached information across GPU, CPU, and storage resources; batch inference optimization; prefill/decode disaggregation, which separates reading a prompt from generating a response; and wide expert parallelism, which distributes large mixture-of-experts models across many accelerators. Those techniques predate version 1.0; according to IBM, what is new is the integration and operational maturity around them, the day 2 operations discipline of holding performance when traffic changes, hardware fails, software is updated, or different users compete for the same resources.

Within 1.0, flow control, fairness, and prioritization are meant to keep bursty workloads from overwhelming latency-sensitive services, whether the workloads share the same hardware or span a mix of accelerators, and requests can remain pinned to the appropriate infrastructure across multiple turns to keep prefix caches warm. llm-d works with inference engines such as vLLM or SGLang, adding distributed routing and operational capabilities on top. Stephen Watt, vice president in Red Hat’s office of the chief technology officer, said llm-d unifies the PyTorch and CNCF ecosystems by bringing Kubernetes-native orchestration directly to vLLM and modern inference engines.

Demonstrated Results and Stated Costs

In a benchmark post published July 22, 2026, the project measured agentic serving of GLM-5.2, described there as a 744B-parameter mixture-of-experts model with 39B active parameters and 78 layers. The workload analysis drew on 219 production Claude Code sessions in which the median main-agent request carried 195K input tokens and returned 317 output tokens, and 96% of main-agent turns reused at least 90% of their input. The post reports that prefix-aware routing captured 96–97% of the cache reuse the workload made available. At an assumed H200 rental rate of $2.71 per GPU-hour, it puts self-hosted serving on two 64-GPU layouts at $0.05 to $0.08 per million input tokens and $1.92 to $3.55 per million output tokens, figures the authors state cover GPU rental only and assume sustained load.

In a separate demonstration, IBM and Red Hat deployed GLM-5.2, characterized in IBM Research’s account as an approximately 753-billion-parameter open-weight model with about 39 billion active parameters, on 544 NVIDIA H100 GPUs. IBM reports that the deployment delivered more than 6.6 million output tokens per minute at peak while serving up to 3,000 concurrent coding agents with zero preemptions, and that a 2,500-agent operating point sustained 7,612 requests per minute. IBM states that, at current cloud rental rates, self-hosting GLM-5.2 on H100 GPUs with llm-d costs 5 to 10 times less per token than equivalent commercial API pricing. “Serving thousands of concurrent agents without preemption gives operators a credible path from a benchmark result to a production deployment,” said Maroon Ayoub, a senior principal machine learning engineer on Red Hat’s inference engineering team.

Underneath those results, according to the same account, the disaggregated data plane completed 6.2 million KV-cache transfers averaging 2.71 GiB per transfer with zero transfer failures, and all benchmarks completed with zero serving errors.