Cloud & Compute Infrastructure

Google Cloud made Managed Service for Apache Spark 4.0 generally available, supporting both serverless deployments and managed clusters.

· ComputeLabs Research · from the November 20, 2026 edition

Google Cloud announced general availability of Managed Service for Apache Spark 4.0, with a runtime standardized on Apache Spark 4.2 for both serverless and managed-cluster deployments. The package includes Java 25, Scala 2.13, Python 3.13 with Pixi, Apache Iceberg 1.12, and Delta Lake 4.4, while Spark 4.2 includes Spark Declarative Pipelines and Real-Time Mode.

Google said customer use of its serverless Spark deployments had nearly doubled over the preceding year, with millions of jobs running monthly. The service operates on Google Cloud’s lakehouse, allowing Spark and BigQuery to read and write the same tables, while Knowledge Catalog provides shared governance, policies, lineage, and semantic discovery for AI agents.

The redesigned console adds an Overview page with samples and public lakehouse datasets, one-click Visual Studio Code authoring in Cloud Shell, a unified Jobs view, and an AI agent for diagnosing job failures. The Data Agent Kit configures Model Context Protocol (MCP) servers, skills, and application programming interface (API) access for building and running pipelines from Visual Studio Code, Claude Code, and Antigravity.

Default Gen4 infrastructure provides faster startups at no additional cost, while Google claims Lightning Engine offers up to 3.25 times the price-performance of the leading high-speed Spark alternative. The release also introduces unified Data Compute Unit (DCU) quotas, serverless infrastructure fungibility, and automatic attachment to the Lakehouse runtime catalog rather than a legacy Hive Metastore; the performance figure is a vendor claim, not an independent measurement in the supplied material.

  • Google Cloud

All 19 stories from November 20, 2026