Google’s multi-cluster inference gateway added less than 1% routing overhead in a benchmark spanning 17,000 compute nodes across three clusters.

Google benchmarked its multi-cluster GKE Inference Gateway across 17,000 compute nodes in the United States and Europe. The deployment used three GKE clusters in `us-east5`, `us-west8`, and `europe-west4`, with `us-east5` serving as the configuration cluster.

The benchmark served a leading mixture-of-experts (MoE) foundation model using SGLang, although the supplied excerpt does not name the model. Google reports near-linear throughput scaling across three clusters, a 99.9% success rate under heavy multi-client concurrency, and gateway throughput equal to 99.5% of a direct local-cluster call.

The architecture places global, multi-region traffic distribution and high availability in the gateway, with memory-aware scheduling handled by the LLM-d router beneath it. An Endpoint Picker Proxy (EPP) reads key-value (KV) cache token utilization from inference engines and exposes a metric used to redirect traffic from a heavily utilized region to another healthy region.

Requests enter through one global virtual Internet Protocol (IP) address, and Google describes the architecture as runtime-, model-, and accelerator-agnostic, supporting GPUs and tensor processing units (TPUs). The 17,000-node figure is a compute-node count—not a disclosed GPU or TPU count—and the reported overhead is a benchmark-specific throughput comparison.