Attacker web pages are in 31% of answers from AI search tools

Researchers examined data-void attacks on AI tools that search the web. A data void is a topic with a small number of pages that have correct content. An attacker puts pages that it controls into it, and the AI tool then uses them. In all test conditions, the attacker content was in 31% of answers. The tests used ChatGPT, Gemini and Sonar Pro.

How it works

The attacker makes pages for topics where only a small quantity of correct content is available. The AI tool searches the web and finds that page. It uses the page or shows it as a source in its answer. The user sees no ranking cues and no other results, and cannot see that the source is weak.

The numbers

The study changed the tone of the prompt (skeptical, excited or neutral), the model and three attacker controls. A skeptical prompt decreased success by a small quantity and increased warnings by a small quantity. The authors found no large effect of tone or model, but they report that small effects are possible.

What is not known

The authors did not examine a defense. They report that defenses are necessary. The study includes three models only.

Insight: Attackers reach AI through what it reads and operates

The authors of a different study report that a backdoor word made one model show brand content in 55% of outputs. Attackers also changed the release pipeline of LiteLLM, which has about 23,000,000 downloads each week. Attackers work on the items that an AI reads or operates, such as web pages and software packages, and the model owner does not control these items. Checks must include these inputs, not only the model.