Data leak

How a data leak happens in an AI system

A data leak in an AI system goes through one of four paths. Each path has a public incident.

Path	What leaks	Incident
Open storage	Datasets, logs, keys	Microsoft AI researchers, 38 TB, disclosed Sep 2023
Data that users send	Source code	Samsung engineer and ChatGPT, reported May 2023
Bug in the AI product	Chat titles, payment data	ChatGPT, redis-py bug, 20 Mar 2023
Model output	Training data, context data	ChatGPT divergence attack, Nov 2023

The model output path has its own pages. A model can give again what it learned, which is memorization. A model can also give data from its context, which is LLM data leakage.

Why a data leak is a risk in AI security

The other three paths do not go through the model. A filter on the model output does not stop them.

In practice

Open storage. Wiz found a URL in a GitHub repository of a Microsoft AI research team. The URL gave access to a full Azure storage account. It held secrets, private keys, passwords and more than 30,000 internal Teams messages. The token gave “full control”, not read only access.

In January 2025, Wiz found a DeepSeek database open with no authentication. It held more than 1 million log entries with chat history and secret keys.

Data that users send. A Samsung engineer uploaded internal source code to ChatGPT. Samsung then banned AI chatbots for staff. The problem: the data stays on the servers of the vendor, “with no easy way to access and delete” it.

Bug in the AI product. OpenAI said a bug in the redis-py library let some ChatGPT users see the chat titles of other users. Payment data of about 1.2% of ChatGPT Plus subscribers in a nine hour window was possibly shown to other users.

An attacker can also cause the leak. EchoLeak (CVE-2025-32711, CVSS 9.3) let an attacker make Microsoft 365 Copilot disclose data with no user step. Microsoft fixed it, and there was no sign of attacks in the wild. That incident is an LLM scope violation caused by indirect prompt injection.

Controls for each path

Path	Controls
Open storage	Private buckets, short lived and scoped tokens, secret scans of code repositories
Data that users send	Approved AI tools, a clear policy, data loss prevention on uploads
Bug in the AI product	Vendor checks, data retention limits, opt out of training
Model output	Remove secrets and PII before training, differential privacy, least privilege for the model

OWASP lists these risks as LLM02:2025, sensitive information disclosure. Its controls include sanitization of training data, least privilege access, differential privacy and user education.

Where definitions disagree

The term has two meanings.

• Security sense. The NICCS glossary of CISA defines data leakage as “the accidental exposure of sensitive corporate or personal data”. It keeps “data theft” for deliberate theft. NIST puts the two in one class, data loss.
• Machine learning sense. The same NICCS entry also uses the term for a different problem. Future or test set data goes into model training by mistake. In this sense, no outsider sees the data.

The line between leak and breach is also not sharp. EchoLeak needed an attacker, but its name says leak. When a report says “data leak”, find out if an attacker was involved.

Questions and answers

What is the difference between a data leak and a data breach?

A data leak is accidental exposure, for example an open storage bucket. A data breach usually means that an attacker got in on purpose. The NICCS glossary of CISA defines leakage as "accidental" and theft as "deliberate". NIST puts the two in one class, data loss.

Can an AI model leak its training data?

Yes. A model can memorize parts of its training data and give them again. In 2023, Nasr and colleagues extracted gigabytes of training data from open models. Their attack made ChatGPT give training data 150 times more frequently than in usual use.

How do companies stop AI data leaks?

Many of the controls are standard security controls. Lock the storage of datasets and logs. Give the AI system least privilege. Let staff use only approved AI tools. Remove secrets and personal data before training. OWASP also lists differential privacy and redaction of sensitive data.