A study finds 57.8% of agent tasks marked done had no proof

Researchers published Soei, a system that records the work of AI coding agents. They tested a prototype on 45 sessions of coding agents. They estimate that 57.8% of tasks that agents marked complete after an edit had no proof in between. The authors say a claim that work is complete must stay apart from proof.

How it works

Soei keeps a local ledger of records. The ledger is a chain. A change to a used entry shows. It holds no content and no credentials. A check layer keeps a claim of “done” apart from proof.

The numbers

• The test used 45 sessions of coding agents.
• 23.4% of edit episodes had no other check (range 17.1 to 35.7).
• 42% of sessions last more than 24 hours.
• 1,326 records held strings that can be credentials.
• The ledger holds 20% of the raw data and detects tampering.

What is not known

The range for the 23.4% is wide, and all results come from one data set of the authors. The authors also checked an open-source tool, Agenttrail. They found a local part that accepts changes with no login.

Insight: Proof of agent work is becoming its own product

Soei, Action State and Arize each make a record of the work that agents did. Soei keeps a claim apart from proof, Action State says that records must be free to check, and Arize scores each step. This reads as a race to control the record of agent work. The record that teams use first can set which items are proof. Leaders must tell the vendor which record they can show.