Token

A token is the fundamental unit of input and output for a large language model. Rather than processing text character by character or word by word, language models use a tokenizer to break text into tokens according to a learned vocabulary. Depending on the tokenization scheme, a token might be an entire common word ('hello'), a word fragment ('un' + 'believ' + 'able'), a punctuation mark, or a special control character.

The most widely used tokenization algorithm is Byte Pair Encoding (BPE), which starts with individual characters and iteratively merges the most frequent character pairs until a target vocabulary size is reached. This produces a vocabulary where common words are represented as single tokens and rare words are decomposed into subword fragments. GPT-4 uses a vocabulary of about 100,000 tokens; the average English word corresponds to roughly 1.3 tokens.

The token count of a piece of text determines the computational cost of processing it and whether it fits within the model's context window. Context windows are measured in tokens: a model with a 128,000-token context window can process roughly 100,000 words of English text simultaneously. Long documents, multi-turn conversations, and code files with large codebases consume context quickly.

Tokenization also explains some model failure modes. Tokenization of words in unexpected ways (especially unusual proper nouns, code, or numbers) can cause the model to process them differently than intended. Arithmetic on large numbers is difficult partly because each digit may be a separate token, breaking numerical structure that would be obvious to a human reader.

For AI trainers, token limits affect how much context can be included in a task, and extremely long tasks may require summarization of earlier turns.