Differential privacy

NIST SP 800-226 puts the promise of differential privacy in one line: “the chance of an outcome is about the same whether or not an individual contributes their data.”

How it works

The definition compares two datasets that differ in one person’s data. NIST calls them neighbouring datasets. For every possible result, the probability from one dataset may differ from the other by at most a fixed factor set by ε, for every person.

“One person” is a choice, which NIST calls the unit of privacy. Protecting one purchase is weaker than protecting everything a customer ever bought.

The usual way to meet it is random noise. A count of 632 sales might be released as 629 or 636. More noise gives more privacy and less accuracy. In NIST’s words, “Smaller ε means stronger privacy but lower accuracy.”

Releases add up: NIST’s example is that two releases at ε of 1 have a combined ε of 2.

For a model, training is the analysis. The most common method is DP-SGD (differentially private stochastic gradient descent), from Abadi and colleagues (2016). At every training step it caps how far any one example can push the model, then adds noise.

Why it matters

Deep neural networks often memorize their training data, NIST notes, and membership inference exploits that. NIST AI 100-2e2025 states that, by definition, differential privacy mitigates membership inference and data reconstruction, meaning rebuilding training records from a model. Its definition “immediately implies an upper bound” on membership-inference success. How strong that bound is depends on ε.

Differential privacy gives no guarantee against model extraction, because it protects the data, not the model. It does not stop inferences that could be made without a person’s data, such as applying a population fact to them. And a breach of the raw records voids the guarantee for those records.

Trade-offs

Accuracy. NIST states that adding differential privacy to training typically lowers accuracy, sometimes significantly. It works best for simple models and very large datasets, and pre-training on public data helps. Abadi and colleagues tested two standard image benchmarks. At ε = 8, with δ (a small allowed chance of leaking) at 10^-5, accuracy fell from 98.3% to 97% on MNIST handwritten digits, and from about 80% to 73% on CIFAR-10, where both models reused layers pre-trained on a public dataset. At ε = 0.5, MNIST fell to 90%.

Choosing ε. NIST SP 800-226 offers no fixed value. It cites one study suggesting ε of 0.1 generally gives strong protection and that values below 1 are considered reasonable. Many deployments have used values between 1 and 20, and NIST warns that “Large values of ε may not provide meaningful privacy.” One study it cites showed significant leakage from private neural networks at ε = 10 when the training data was maliciously crafted. The 2020 U.S. Census release used ε = 19.61, which NIST AI 100-2e2025 gives as an example of large parameters being common in practice, because the worst-case maths is often not tight. NIST SP 800-226 adds that larger values “may still provide meaningful privacy in some cases”. NIST AI 100-2e2025 recommends empirical privacy auditing alongside the maths.

Whom you trust. In the central model a trusted curator holds the raw data and adds noise to results. In the local model each person adds noise before sending anything, which is why NIST says Google’s RAPPOR system and Apple’s data collection used it. The price is accuracy: NIST states the local model is typically not used for complex applications like machine learning.

Questions and answers

What is differential privacy in simple terms?

Differential privacy is a promise that the result of an analysis comes out about the same whether or not any one person's data was included. Because one person's data barely changes the result, the result reveals little about that person. The promise is usually kept by adding calibrated random noise.

What does epsilon mean in differential privacy?

Epsilon (ε) is the privacy parameter, also called the privacy loss or privacy budget. It bounds how much the chance of any result may change when one person's data changes. In NIST's words, "Smaller ε means stronger privacy but lower accuracy." NIST offers no fixed value and warns that large values may not provide meaningful privacy.

Does differential privacy protect a machine learning model from attacks?

It protects the training data, not the model. NIST states that differential privacy mitigates membership inference and data reconstruction by definition, but gives no guarantee against model extraction, and several papers reported negative results using it against property inference.

Does differential privacy make a model less accurate?

Usually, yes. NIST states that adding differential privacy to training typically lowers accuracy, sometimes significantly, and works best for simple models and very large datasets. In Abadi and colleagues' 2016 experiments at ε = 8, image-classification accuracy fell from 98.3% to 97% on MNIST and from about 80% to 73% on CIFAR-10 (both CIFAR-10 models reused layers pre-trained on a public dataset), and fell further at smaller ε.