Google makes Managed Spark 4.0 available to all users
Google tells that version 4.0 adds an AI agent that finds the cause of a data task error and gives a repair.
Google Cloud made Managed Service for Apache Spark 4.0 available to all users. Apache Spark is open-source software that does large data tasks. In the serverless type, Google operates the computers. Google tells that customers use it almost 2 times as much as one year before, with millions of tasks each month. Version 4.0 is for AI agents, which are software that does tasks, and which start many short tasks.
What is new
- A new runtime, the software that operates the tasks, with Apache Spark 4.2, Java 25 and Python 3.13.
- An AI agent that reads logs and query plans and gives a repair.
- A Data Agent Kit that sets up agent tools in VS Code, Claude Code and Antigravity.
- Less time to start on the Premium tier, at no added cost.
The numbers
Google tells that its Lightning Engine gives a maximum of 3.25 times the price-performance of the primary fast Spark alternative. Google did not tell how it did the test, and did not name that alternative.
What is not known
Runtime profiles are in preview. Google gave no test results for the repair agent or for the time to start.
Insight: Google makes one agent route through its data tools
On 4 November Google made Knowledge Catalog the default in Gemini, and agents start from governed business definitions. On 12 November it opened an MCP server, a standard connection for agent tools, for cloud policies. This reads as a plan to supply data, policies and computers to each agent, also an agent from a different firm. The firm that supplies all three can control how agents work.
Sources
- What’s new in Managed Service for Apache Spark 4.0cloud.google.com
Posted