Token

How it works

Before a language model sees any text, a tokenizer splits it into tokens from a fixed vocabulary. The model works on those tokens, not on the raw text.

Many tokenizers split rare words into pieces. Byte pair encoding (BPE), adapted for subword splitting by Sennrich and colleagues in 2015, starts from single characters and repeatedly merges the most frequent adjacent pair. Common words end up as one token; a rare word becomes several. Anthropic’s glossary states the trade-off: larger tokens are efficient, smaller ones let a model handle words it has never seen.

A context window is sized in tokens. Some APIs also price by them: Anthropic charges per million input tokens and per million output tokens.

Why it matters

Token counts decide what a request costs and whether it fits. MITRE ATLAS advises generative AI services to limit context and output tokens, one of its defences against cost harvesting. Counts also vary more than people expect. A 2023 Oxford study found the tokenizer used by ChatGPT and GPT-4 needed up to 15 times as many tokens for the same text in Shan as in English, and 3 times in Arabic. Anthropic says Claude 4.7 and later models produce about 30% more tokens than earlier ones for the same text, with the exact increase depending on the content.

The tokenizer is also an attack surface. In 2025 HiddenLayer’s TokenBreak changed words slightly, so that AI guardrails classifiers using BPE or WordPiece, a similar method, missed a prompt injection the target model still understood. Classifiers using a different method, Unigram, were not fooled in those tests. Separately, tokens that a model barely saw in training, called glitch tokens, can make it behave unpredictably; a 2024 Cohere study found them in every model it tested.

Questions and answers

How long is a token?

It depends on the tokenizer and the language. Anthropic's glossary puts a Claude token at about 3.5 English characters, varying by language. Other languages can need more tokens for the same text: a 2023 Oxford study found the tokenizer used by ChatGPT and GPT-4 needed about 1.6 times as many for Italian as for English, 3 times for Arabic and up to 15 times for Shan.

How do tokens affect what an AI API costs?

Commercial AI services charge per token or per character, a 2023 Oxford study notes. Anthropic, for example, prices each model per million input tokens and per million output tokens. A tokenizer change alters the count for the same text: Anthropic says Claude 4.7 and later models produce about 30% more tokens than earlier ones, depending on the content.

What is a glitch token?

A glitch token is a token in a tokenizer's vocabulary that the model barely saw in training, such as _SolidGoldMagikarp. Prompts containing one can make the model behave oddly. A 2024 Cohere study found such tokens in every model it tested.