What matters in AI.

Subscribe

Learn / AI basics

Definition · AI basics

LoRA adapter

A LoRA adapter is a small set of trained weights that changes the behavior of a base AI model. The base weights stay frozen. The adapter adds two small low-rank matrices to selected layers. The adapter file is much smaller than the base model. The base model is necessary to load the adapter.

Last reviewed

Key points

  • A LoRA adapter trains two small matrices for each selected layer and keeps the base model weights frozen.
  • On GPT-3 175B, Hu and colleagues cut trainable parameters by 10,000 times and the checkpoint from 350GB to 35MB (2021).
  • The adapter file holds only the adapter weights. The base model is necessary to load it, and the config names that base model.
  • Teams can merge an adapter into the base weights, or keep many adapters and change them for each task.
  • A shared adapter is untrusted code for your model. Research shows that an adapter can remove safety training or contain a hidden backdoor.

How a LoRA adapter works

A LoRA adapter does not change the base weights. It adds a small correction to the output of each selected layer.

  1. Freeze the base. The base model weights (W0) stay the same. See frozen model.
  2. Add two small matrices. For each selected layer, LoRA adds a matrix A and a matrix B. Their inner dimension is the rank r, for example 4, 8 or 16.
  3. Train only A and B. B starts at zero. Thus the model starts with no change.
  4. Apply the result. The layer output is W0x + BAx. BA has the same shape as W0, but A and B hold far fewer parameters.
A LoRA layer: the frozen base weight and the small adapter path both read the input, and their outputs are added.
A LoRA layer: the frozen base weight and the small adapter path both read the input, and their outputs are added.

PEFT says that a higher rank gives “more learning capacity”. PEFT also says that LoRA is usually applied only to the attention blocks of a transformer. Other layers can also be targets.

Why LoRA adapters matter for security

A LoRA adapter is a small file that changes what a model does. Examine it as carefully as a new model.

Four risks of LoRA adapters:

  • Safety training removed. Lermen and colleagues used LoRA, one GPU and less than $200 on Llama 2-Chat 70B. Its refusal rate fell to about 1% on two benchmarks (2023).
  • A hidden backdoor. Liu and colleagues trained a backdoor LoRA one time. They merged it into other task adapters, which kept the backdoor and their useful skills.
  • Dangerous file format. PEFT says that the older adapter_model.bin uses pickle and has “security vulnerabilities”. The default is safetensors.
  • Runtime load on a server. vLLM says that runtime load of adapters “comes with security risks”.

In practice

These numbers show the scale of LoRA.

Example Base model Adapter Source
GPT-3 175B, r=4, query and value only 350GB checkpoint 35MB Hu et al., 2021
GPT-3 175B, training memory 1.2TB without LoRA 350GB with LoRA Hu et al., 2021
Apple on-device model, about 3B parameters Not given Tens of MB at rank 16 Apple, June 2024
QLoRA, 65B model 4-bit frozen base Trained on one 48GB GPU Dettmers et al., 2023

After training, a team has two alternatives.

Alternative What you keep Advantage Cost
Adapter not merged Base model plus adapter files Add or remove adapters for each task Possible latency (PEFT)
Merged model One full set of weights No added latency (Hu et al.) Cannot unmerge or change adapters, much larger file (PEFT)

The PEFT library saves three files for an adapter:

  • adapter_model.safetensors (or .bin): the adapter weights only.
  • adapter_config.json: the rank, the target layers and the base model name.
  • README.md: a model card. PEFT does not use it to load the adapter.

Server software can keep one base model and many adapters. vLLM serves adapters “on a per-request basis with minimal overhead”. Apple loads, caches and swaps adapters on its devices. The LoRA paper gives one limit: one forward pass cannot easily batch inputs for different adapters.

Checks before a team loads a shared adapter:

  1. Make sure that adapter_config.json names your base model and its revision. See model provenance.
  2. Reject .bin files from an unknown source. Load safetensors only.
  3. Examine the hash or signature of the file. See verify AI artifacts.
  4. Test the adapted model again for refusals before release. Test results of the base model do not apply to it.

vLLM says to use runtime load of adapters only in “an isolated, fully trusted environment”.

Where definitions disagree

The word “adapter” has two senses.

  • Adapter layers. Houlsby and colleagues (2019) added new small layers into the network. On GLUE, their performance was 0.4% or less below the performance of full fine tuning, with 3.6% added parameters for each task. The LoRA paper uses “adapters” for this method. It says that LoRA, “unlike adapters”, adds no inference latency.
  • A small add on. Hugging Face PEFT and Apple use “adapter” for a LoRA module. In this sense, “adapter” is a small trained part on a frozen base.

When a paper compares LoRA with “adapters”, it uses the first sense.

Questions and answers

Does a LoRA adapter need its base model?

Yes. The adapter file holds only the adapter weights, not the base model. The Hugging Face PEFT documentation says that the initial model is necessary to load an adapter. The file adapter_config.json names that base model in the field base_model_name_or_path.

Is a LoRA adapter in safetensors format safe to load?

Safetensors removes the code risk of pickle files. PEFT says that the older .bin format has this risk. The format does not examine the weights. Liu and colleagues show that a LoRA adapter can contain a hidden backdoor.

What is the difference between LoRA and QLoRA?

LoRA trains small adapter matrices on a frozen base model. QLoRA does the same on a base model quantized to 4 bits, which needs less memory. Dettmers and colleagues fine-tuned a 65B model on one 48GB GPU with QLoRA (2023).

Sources

  1. LoRA: Low-Rank Adaptation of Large Language ModelsarXiv (Hu et al., Microsoft), 16 Oct 2021
  2. LoRA: Low-Rank Adaptation of Large Language Models (section 4.1)arXiv (Hu et al., Microsoft), 16 Oct 2021
  3. LoRA (PEFT developer guide)Hugging Face
  4. PEFT checkpoint formatHugging Face
  5. QLoRA: Efficient Finetuning of Quantized LLMsarXiv (Dettmers et al.), 23 May 2023
  6. Introducing Apple's On-Device and Server Foundation ModelsApple, 10 Jun 2024
  7. LoRA Adapters (vLLM documentation)vLLM project
  8. LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70BarXiv (Lermen, Rogers-Smith and Ladish), 22 May 2024
  9. LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play EcosystemarXiv (Liu et al.), 30 Apr 2025
  10. Parameter-Efficient Transfer Learning for NLParXiv (Houlsby et al.), 13 Jun 2019

In the news