Google Kubernetes Engine Pod snapshots reduced inference startup latency by up to 89% in benchmarks, loading 70-billion-parameter models in 37 seconds.

Google Kubernetes Engine (GKE) Pod snapshots save a workload’s running state, including central processing unit (CPU) and GPU memory, and restore it on demand. Google reports startup-latency reductions of up to 89%, with 70-billion-parameter models loading in 37 seconds and 8-billion-parameter models in 15 seconds.

The mechanism performs workload initialization once and stores the fully loaded state in high-throughput Cloud Storage. New replicas then restore that state rather than independently repeating server initialization, model-weight downloads, and loading into accelerator memory.

Google identifies llama3-70b among the large models used in its benchmarks and says conventional initialization can take several minutes. These figures measure startup, not token-generation latency or steady-state inference throughput, and the supplied excerpt does not quantify realized customer cost savings.

The feature also addresses agentic workloads requiring isolated sandboxes for code execution or computer-use tools that start quickly and suspend when idle. Google additionally identifies game servers and complex Java applications as workloads affected by lengthy initialization, although the supplied quantitative results concern AI models.