A copy guard cut copying to 0%, but an attacker found 48% of facts

Researchers examined 118 papers on defenses for retrieval-augmented generation (RAG), where an AI system looks up documents before it answers. The papers use different metrics, datasets and threat models, and thus the results cannot be compared. In a test by the authors, a guard that stops copied text decreased 20-token copying to 0.0%. A three-query attacker found 48.3% of hidden facts on Qwen3-8B.

How it works

The study puts papers in groups by the endpoint, which is the item that they measure, and not by the stated goal. The testbed adds a copy guard. Then it uses an attacker with queries that change after each answer.

The numbers

• Only 17.8% of studies use content disclosure or reconstruction as the primary endpoint.
• For copyright studies, 73.3% of records measure corpus-use detection. None measure emitted expression.
• Thirty-two annotators examined each paper two times.

What is not known

Most tests use general question sets and attacks that do not change. The risk for private data is thus not known. The testbed result is for one model, Qwen3-8B.

Insight: Defenses are tested on the wrong risk

A different study reports that RAG precision decreased by 63% when a Wyoming system increased from 54 to 1,128 documents. In this study, most tests use general data and weak attacks. Defenders measure the items that are easy to measure, such as copying, and leaks go around them. A vendor report that it stops leaks is not sufficient without the metric behind it.