llm-d 1.0 launched as an open-source, Kubernetes-native distributed inference stack, adding priority routing, session-aware scheduling, and zero-downtime updates.

The release coordinates multiple model servers as a distributed serving pool. Google describes llm-d as an open-source, Kubernetes-native stack for large language model inference, co-founded with Red Hat, IBM, and others. The Cloud Native Computing Foundation (CNCF) Sandbox project has more than 500 contributors across more than 25 organizations.

Version 1.0 adds explicit controls for competing workloads and agent sessions. Features include priority- and fairness-aware routing, session-aware routing, turn-priority scheduling, and native Responses API state. Earlier capabilities include prefix- and load-aware routing, key-value (KV) cache offloading, separation of prompt processing from token generation, wide expert parallelism, and batch inference.

Operational features address upgrades and request handling. The release provides v1 application programming interfaces, a hardened request path validated across multiple clouds and accelerator types, router high availability, and zero-downtime updates. Its DisaggregatedSet component coordinates rolling updates around separate prompt-processing and token-generation workloads without dropping context or disrupting active sessions, according to Google.

The conference demonstration used a mixed accelerator fleet. Google showed 1,000 sandboxed agents on Agent Substrate on Google Kubernetes Engine (GKE), accessing one llm-d endpoint across vLLM, SGLang, TPUs, and GPUs. The release also adds native PyTorch serving through TorchTPU cooperative time-slicing; Google describes throughput and utilization benefits but provides no numerical benchmark in the supplied excerpt.