CarryoverBench shows that a planted attack fires again in 5 rounds

Researchers made CarryoverBench, a test set of 12 personal-assistant scenarios and 101 attack tasks. Agents keep data across sessions in 2 paths. One path is memory files that go into the system prompt. The other is scheduled tasks that start the agent with no user input. A prompt injection is text that tells a model to do a step. It can be kept in one path and fire again when the attacker is not there.

How it works

The test has 3 paths to start the payload:

• Instant: the prompt injection fires in the same turn.
• Autonomous: a scheduled task starts the payload.
• Latent: the memory makes the payload start again.

A rule-based verifier scores planting and the start as 2 different steps.

The numbers

In 5 frontier models, planted payloads fired again with no decrease that the researchers could measure in 5 rounds. When the planting step was done, the start step gave small protection.

What is not known

The researchers moved the Latent path to a different runtime, Hermes. The effect was less but did not stop. They write that defense must have controls at the runtime layer and also steps in the system prompt.

Insight: One session score hides the risk that comes after

The researchers write that other prompt injection tests score only the session where the text is first seen. A test of 975 task pairs found that a local attack that works does not always cause damage. Together these facts read as a sign that a one session score is weak. The damage can come in a session after it, and no test examines that. A clean test result does not show a clean agent memory.