llm-d 1.0 ships a stable, Kubernetes-native stack for distributed LLM inference

20 Oct 2026Big news

The CNCF llm-d project hit 1.0, adding a stable v1 config API, a highly available router, priority and fairness flow control, session-aware routing for multi-turn agents, and prefill/decode pools managed as one unit. It sits in front of vLLM or SGLang and keeps prefix-aware routing, KV-cache offload to host memory or SSD, and wide expert parallelism.

How to try it

llm-d is a router and orchestration layer that spreads requests across vLLM or SGLang replicas in a Kubernetes cluster, routing by cached prefix and load. The quickstart deploys the router in standalone mode plus two vLLM replicas of Qwen3-32B on NVIDIA GPUs.

1. Install kubectl and helm, then clone the repo and check out the release-1.0 branch.
2. Source guides/env.sh, set GUIDE_NAME and NAMESPACE, install the Gateway API Inference Extension CRDs and create the namespace.
3. Create the llm-d-hf-token secret with your Hugging Face token.
4. Helm-install the standalone router with the optimized-baseline values, then apply the vLLM model server kustomization.
5. Get the router ClusterIP and send a request to /v1/completions from a debug pod.

export branch="release-1.0"
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}
export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
source ${REPO_ROOT}/guides/env.sh
export GUIDE_NAME="quickstart"
export NAMESPACE=llm-d-quickstart
helm install ${GUIDE_NAME} \
    ${ROUTER_STANDALONE_CHART} \
    -f guides/recipes/router/base.values.yaml \
    -f guides/optimized-baseline/router/optimized-baseline.values.yaml \
    -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}
kubectl apply -n ${NAMESPACE} -k guides/optimized-baseline/modelserver/gpu/vllm/base/

Open the official docs

Source: Read the full story on Google Cloud Blog

inferencekubernetesvllmkv-cacheopen-source