Transformer (Architecture)

The transformer is a neural network architecture introduced in the 2017 paper 'Attention Is All You Need' by researchers at Google. It has become the dominant architecture for natural language processing and has been adapted for image, audio, and multimodal tasks. Every major large language model in commercial use today, including GPT-4, Claude, Gemini, and Llama, is built on the transformer architecture.

Before the transformer, sequence models like recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) processed text one token at a time, in order. This made them inherently sequential and difficult to parallelize during training, limiting the scale at which they could be effectively trained. The transformer replaced recurrence with the attention mechanism, which allows the model to process all positions in a sequence simultaneously and directly model relationships between any two tokens regardless of their distance in the sequence.

A transformer model consists of stacked layers, each containing a multi-head attention component and a feed-forward network. In each layer, the attention component computes a weighted combination of all positions in the sequence, where the weights reflect how relevant each position is to the current one. The feed-forward component then processes each position independently. These layers are stacked many times, with earlier layers capturing local and syntactic patterns and later layers capturing semantic and long-range dependencies.

The scalability of the transformer architecture is central to the success of large language models. As models grow from millions to hundreds of billions of parameters, and as training datasets grow from gigabytes to trillions of tokens, transformer-based models have continued to improve on nearly every benchmark.