GPT-5.6-Sol gives confidential facts in 7.08% of answers in a test

Researchers did a test of 3 AI agents that read confidential files. They name the risk the mosaic effect: an agent connects small facts from documents, and then tells a confidential result. GPT-5.6-Sol is the top model in the test, but it did this in 7.08% of answers in a simulation, also for a safe task. The authors report that live agents had more leaks.

How it works

The team wrote 30 test examples in 3 fields: cyber security, medicine and military work. In each example, no one document holds the confidential result. The agent must connect facts from some documents.

The numbers

• Simulation: 10,800 answers from 3 agents.
• Live test: 1,440 answers from 2 open source tools.
• GPT-5.6-Sol: 7.08% leaks, with a range of 3.75% to 10.67%.
• With class labels and rules for facts: 32.37% of answers have a leak.

What is not known

The examples are test data, and no other group did the test again. A refusal does not stop the leak, because some refusals tell the confidential result. The paper gives no safe level.

Insight: Two methods each leave about 3 leaks in 10

Labels and rules gave a leak in 32.37% of answers in this test. In a related test, a sanitizer (a filter for data between agents) cut leaks from 96.7% to 29.25%. The tests are different. Do not compare the numbers. A large leak stays after each method, which reads as a sign to limit the files that agents can read. Control access, not only rules.