Declarative pipelines cut cluster time 34 times in an agent test

A test compared declarative and procedural pipelines that an AI agent makes, in Apache Spark. Pipelines are code that moves and changes data. The agent used Opus 4.8 and made 528 pipelines for 22 tasks. Declarative pipelines, where the agent describes the result, used 17 executor-seconds of cluster time. Procedural pipelines used 596. The authors tell that agents make more than 70% of these pipelines on Databricks at this time.

How it works

A procedural pipeline gives each step to Spark. Spark must operate it to find an error. A declarative pipeline describes the datasets that the agent must make. Spark checks each column and each dependency in a dry run before it starts.

The numbers

The median declarative pipeline had 64 lines, against 131. Cluster use was 17 executor-seconds, against 596. Declarative pipelines used about 2.3 times more tokens (the units that a model reads and writes). Some procedural pipelines put all data in memory. This can stop the pipeline when the data increases. No declarative pipeline did this.

What is not known

A dry run cannot find a silent error, where a pipeline operates and gives an incorrect result. The gap between the silent error rates was about 15% or less. Tests and inspection continue to be necessary for logic. The authors give a repository with the data and code, to do the test again.

Insight: Checks before the start are where vendors compete

Buildkite added a tool that checks a pipeline before it starts. The data agent of IBM shows a plan for approval before it operates. The vendors gave no test results. This reads as a race to control the check, and the vendor with test numbers possibly sets the standard. Leaders can tell each vendor to show such results.