Memorization

How memorization is measured

Researchers test what the language model gives back. The usual test is from Carlini and colleagues (2022):

1. Get a text from the training data.
2. Give the first k tokens of the text to the model as a prompt.
3. Let the model write the next tokens. It selects the token with the highest probability each time.
4. Compare the output with the remaining part of the text. If the two are the same, the text is memorized.

An attacker with no copy of the training data does the steps in the opposite sequence. The attacker gets much text from the model, then finds which outputs are training text.

What memorization lets an attacker do

Memorization lets an attacker get training data with queries only. Researchers show four risks:

• Personal data. An attack on GPT-2 got names, phone numbers and email addresses. In 2023, 16.9% of tested ChatGPT outputs from one attack had memorized personal data.
• Secrets. The 2018 “Secret Sharer” study got secret text “such as credit card numbers” from models trained on it.
• Books. In 2025, Llama 3.1 70B “entirely memorizes some books, like Harry Potter”. Most of the 14 open-weight models tested did not memorize most of the 200 books.
• Test scores. On GSM1k, a new set of school maths problems, scores were a maximum of 8% lower than on GSM8k (Zhang et al., 2024).

What makes a model memorize more

Carlini and colleagues (2022) found three causes. Each one has a log-linear relation to memorization.

Cause	Result
Model size	In GPT-Neo, 10 times more parameters gave 19 percentage points more memorization
Copies of a text in the data	More copies, more memorization
Prompt length	A longer prompt gets more memorized text

Copies are not necessary. The GPT-2 study got texts that were “included in just one document” of the training data.

4 methods to decrease memorization, and their limits

Method	What it does	Limit
Deduplication	Removes copies of the same text from the training data	10 times less memorized output (Lee et al., 2021), but it “will not perfectly prevent leakage”
Sanitize training data	Removes sensitive data before training	“some private data will always slip through” (Carlini et al., 2021)
Differential privacy	Gives a formal limit for each training record	The model can learn less from the rare examples, the “long tails”
Memorization tests	Measure the risk, as Google did for Smart Compose	Show the problem, do not remove it

OWASP gives data sanitization and differential privacy as mitigations for LLM02:2025 Sensitive Information Disclosure. A membership inference attack is a related risk: it finds if a record was in the training data.

Where definitions disagree

A fault or a necessity. Security papers see memorization as a leak. Carlini and colleagues say that it “violates privacy”. Feldman (2019) proves that memorization “is necessary” for good results when the data has a long tail of rare examples.

Training text or test answers. Privacy research means text that the model gives back word for word. Benchmark reports use the word for a model that learned a test set. The GSM1k authors write that “some models may have partially memorized GSM8k”. They also write that “many models, especially those on the frontier, show minimal signs of overfitting”.

When a report says that a model shows signs of memorization, find which sense it uses.

Questions and answers

How much of its training data does a language model memorize?

Carlini and colleagues (2022) found that the 6 billion parameter GPT-J model memorizes at least 1% of its training data, The Pile. In the GPT-Neo family, a model 10 times larger memorized 19 percentage points more. The authors write that memorization "will likely get worse" as models get larger, if there are no active mitigations.

Does safety training stop a model from giving out memorized data?

No. In 2023, Nasr and colleagues made ChatGPT (gpt-3.5-turbo) give out training data at 150 times the usual rate. They asked it to repeat one word with no end. Their result is that "current alignment techniques do not eliminate memorization".

How can a team decrease memorization in a model it trains?

Remove copies of the same text, remove sensitive data before training, and think about differentially private training. In one study, deduplication made models give out memorized text 10 times less frequently. No one method stops all leaks.