About ten attention heads cause much of the loss from junk text

Researchers traced why language models decrease in precision when an input holds irrelevant text. This noise is in systems that get documents, use tools or read long inputs. The authors found a path that they name the Intrusion Pathway. They found part of the damage in a small set of attention heads (parts of the model that select which words to use).

How it works

Some heads copy content from earlier text to the position where the model selects its answer. These intrusion heads then promote content from the distractor. They are mostly different from the heads that use related facts.

The numbers

A test that puts back the clean internal condition recovers 97.6% of the drop, on average for four tasks. That test needs a matched clean test, and systems in use cannot make it. A test that turns off about ten heads recovers 57%.

What is not known

The text gives no model names. It says the cost to text that helps is bounded, but gives no figure. The 57% result comes from research and is not a tool for use.

Insight: Junk text can steer a model like a weak attack

An earlier backdoor study used one phrase to make a model send project files out, for less than $50. Here random text, with no attacker, also steers a model through some parts. It seems that the input is a control point: what you put in the text decides how the model acts. Check retrieved text before it goes to the model.