Transformer

How a transformer works

A transformer changes text into tokens, then each token into a vector (an embedding). A stack of layers then changes the vectors again and again. Vaswani and colleagues used 6 layers in the encoder and 6 in the decoder.

Each layer has two parts:

1. Self-attention. Each token makes a query. Each other token gives a key and a value. The query and the key set how much of each value goes into the result.
2. Feed-forward network. A small network changes each token vector independently.

The paper uses 8 attention heads in each layer. Each head can follow a different relation between tokens. Attention does not know the word sequence, thus the model adds position data to each token.

Why the transformer is important for security

1. Instructions and data share one input. The system prompt, the user text and a web page become one sequence of tokens. The UK NCSC (8 December 2025) says LLMs “do not enforce a security boundary between instructions and data inside a prompt”. In an LLM, “there is only ever ‘next token’”. This lets prompt injection work.
2. Long inputs cost more. The 2017 paper gives self-attention a cost for each layer of O(n²·d), for n tokens. Thus each context window has a limit. Very long inputs can also help a denial of AI service.
3. A model does not read a long input equally. Liu and colleagues (2023) found that results are good when the important text is at the start or the end. Results “significantly degrade” when it is in the middle.

Encoder, decoder, or the two together

The 2017 design had an encoder and a decoder, for translation. Many subsequent models keep only one part.

Family	Use	Example
Encoder only	Reads and classifies text	BERT (2018)
Decoder only	Writes text, one token at a time	GPT (OpenAI, 2018), Llama 3 (Meta, 2024)
Encoder and decoder	Changes one text into a different text	the 2017 translation model

A decoder applies a mask. The mask makes sure that the prediction for a position uses only the earlier positions. Thus a chat model writes its answer one token at a time.

In practice: from translation to all AI

The transformer replaced recurrent networks, which read one word after the other. Attention looks at all tokens in one step. Thus training can use many GPUs in parallel.

At this time, the design has many uses, not only translation:

• Text. The largest Llama 3 model from Meta (2024) is a dense transformer with 405B parameters.
• Images. The Vision Transformer (2020) applies it to small patches of an image.

Questions and answers

What does attention mean in a transformer?

Attention is the step where each token gives a weight to each other token. Then it adds their information by those weights. The 2017 paper uses queries, keys and values for this step. It uses 8 attention heads in parallel in each layer.

Is ChatGPT a transformer?

Yes. GPT means Generative Pre-trained Transformer. Hugging Face lists GPT (June 2018) as the first pretrained Transformer model. The GPT family uses the decoder only shape, which is good for text generation.

Why do transformers have a context limit?

Self-attention compares each token with each other token. The 2017 paper gives its cost for each layer as O(n²·d), for n tokens. Two times the input gives about four times the attention work. The largest Llama 3 model from Meta accepts a maximum of 128K tokens.