Contributors to the llm-d project have released llm-d 1.0, the first stable version of an open-source, Kubernetes-native stack for distributed large language model inference — built for the age of AI agents.
Announced at the PyTorch Conference in San Jose, the project is co-maintained by Google Cloud, Red Hat and IBM, with contributions from more than 500 developers across 25-plus organizations including CoreWeave, NVIDIA, AMD, Intel, Cisco, Hugging Face and Mistral AI. It is hosted as a Cloud Native Computing Foundation sandbox project under the Apache 2.0 license.
llm-d 1.0 targets a shift in inference workloads: from simple one-prompt chat to multi-step agents and reinforcement learning rollouts that fan out across tools, share long context and arrive in bursts. It turns a cluster of vLLM or SGLang model servers into one coordinated serving pool, with cache-aware routing, tiered KV cache offloading, prefill/decode disaggregation, multi-tenant flow control and zero-downtime upgrades.
Vendor-reported benchmarks are striking: on an agentic coding workload on Ironwood TPUs, prefix-aware routing cut median time-to-first-token by more than 85% and raised throughput 2.4x versus a plain Kubernetes Service. A keynote demo showed 1,000 sandboxed agents hitting a single llm-d endpoint across a mixed fleet of TPUs and GPUs.
Production users are already on board: Snap reports prefix cache hit rates of up to 75-80%, and Mistral’s inference engineers call it “frontier-class inference performance” without depending on a single proprietary provider.
The same agent workloads llm-d serves are visible across the tool ecosystem, from coding agents like Goose, OpenHands and Cline to the broader wave of autonomous software.
Source: Google Cloud Blog
