Adversarial text attacks

How adversarial text attacks work

An adversarial text attack tries small edits until the model output changes. TextAttack, a Python framework with 16 published attacks, divides each attack into four parts:

1. Goal function: the test of success, such as a changed label.
2. Constraints: the limits on each edit.
3. Transformation: the edits that the attack can try.
4. Search method: the sequence of the tries.

Text is different from images: “the data is continuous but text data is discrete”. Thus, a text attack replaces full units, such as a character or a word.

Why adversarial text attacks matter

Adversarial text attacks can get text through AI filters, such as hate speech detectors and guardrails. The attacker does not change the model.

Test	Population	Result
TextFooler, 2019	BERT, IMDB data	Accuracy fell from 90.9% to 13.6%, with 6.1% of words changed
Gröndahl et al., 2018	7 hate speech models, Google Perspective	Typos, changed word boundaries and added words evaded detection
Bad Characters, 2021	IBM classifier, 250 sentences	Performance fell by a maximum of 75%. Invisible characters had no effect
Hackett et al., 2025	6 AI guardrails	Emoji smuggling reached 100% attack success rate

Four types of adversarial text attack

Type	What changes	Example	Attacker access
Character	One token	HotFlip, 2017	White-box, uses gradients
Word	Words with almost the same meaning	TextFooler, 2019	Black-box, sends queries
Encoding	Invisible characters, homoglyphs, reorderings, deletions	Bad Characters, 2021	Black-box
Added string	A suffix on the prompt	Adversarial suffix, 2023	Result reported on public interfaces of ChatGPT, Bard and Claude

This grouping is our own. A 2020 survey uses other levels: character, word, sentence and multi-level.

A small change can change a result. TextFooler changed “The boys are in band uniforms.” to “The boys are in band garment.” BERT then gave entailment, not contradiction.

In practice: 3 defences and their limits

Defence	What it stops	Limit
Remove invisible characters	Invisible character attacks	Does not stop homoglyphs
Show the text as an image, then read it with OCR	Invisible character, reordering and deletion attacks	Baseline BLEU score fell by 6.2%
Adversarial training	No attack type fully	Preliminary TextFooler test: after-attack accuracy on MR rose from 11.5% to 18.7%

These sources give no complete defence against word attacks.

Where definitions disagree

The term has no fixed limit. Morris and colleagues write that attacks “lack a shared definition of a what constitutes a successful attack”.

• Same meaning is necessary: For Morris and colleagues, an example must fool the model and follow four linguistic constraints. With tighter constraints, attack success on IMDB, Yelp and MR fell by 71 percentage points or more.
• Same appearance is necessary: Bad Characters uses encoding changes. A person sees the same text or almost the same text.
• Only the effect is necessary: Zou and colleagues add a suffix to get objectionable content. The text clearly changes, and the authors use the term adversarial attack.

Before you compare attack success rates, find which definition each paper uses.

Questions and answers

Are adversarial text attacks the same as prompt injection?

No. Prompt injection is one goal, and adversarial text changes are one method. In 2025, Hackett and colleagues used character changes and word changes against six guardrails that look for prompt injection. Changes of this type also work against classifiers with no prompt, such as hate speech detectors.

Does adversarial training stop adversarial text attacks?

Not fully. In a preliminary TextFooler test, adversarial training raised after-attack accuracy on MR from 11.5% to 18.7% only. Gröndahl and colleagues found in 2018 that adversarial training "does not completely mitigate the attacks" on hate speech detectors.

Which adversarial text attacks can input sanitization stop?

Input sanitization can stop some encoding attacks. Bad Characters says that removal of a standard set of invisible characters blocks invisible character attacks. The same paper says that homoglyphs are "perhaps the most challenging technique against which to defend".