Skip to content
Back to the Guavy Wire
Stocks

Google Cloud Unveils Managed Service for Apache Spark 4.0

Instruments
GOOGL
Share

Google Cloud has announced the general availability of Managed Service for Apache Spark 4.0, a major upgrade designed to meet the evolving demands of autonomous AI agents and enterprise data pipelines. The new version introduces a next-generation runtime foundation, optimized for both serverless and managed cluster deployments. Key features include a new 4.0 runtime with Apache Spark 4.2, Java 25, Python 3.13 with Pixi, and Apache Iceberg 1.12, all available for both serverless and managed clusters.

The upgrade also includes a redesigned, code-first Google Cloud console with agentic troubleshooting capabilities. This streamlined console aims to reduce operational overhead by offering frictionless onboarding, optimized defaults, and unified job management. An AI agent can now automatically diagnose job failures, helping engineers quickly resolve issues.

For developers who prefer working within their IDEs, the Data Agent Kit allows them to build and run Apache Spark pipelines directly from VS Code, Claude Code, and Antigravity. The kit configures Model Context Protocol (MCP) servers, skills, and API access, enabling AI agents to generate, tune, and test pipelines without leaving the IDE.

Performance improvements in Managed Service for Apache Spark 4.0 include faster startups at no additional cost, hardware acceleration by default, and the introduction of Lightning Engine. This native C++ vectorized execution engine delivers up to 3.25x the price-performance over leading high-speed Apache Spark alternatives, along with up to 4.9x faster performance compared to open-source Apache Spark.

More on Stocks