Transformer

The transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. Unlike recurrent neural networks that process sequences step by step, transformers use self-attention mechanisms to process all positions of the input simultaneously, enabling massive parallelization during training.

A transformer consists of an encoder and a decoder, each built from stacks of layers containing multi-head self-attention and feed-forward sub-layers. The self-attention mechanism allows each token in a sequence to attend to every other token, capturing long-range dependencies far more effectively than RNNs. Positional encodings are added to the input embeddings to retain information about token order.

Transformer Architecture

The original transformer is an encoder-decoder stack. Token embeddings are combined with positional encodings before entering the encoder. Each encoder layer applies multi-head self-attention, a position-wise feed-forward network, residual connections, and layer normalization. Each decoder layer adds masked self-attention so it cannot read future target tokens, then uses cross-attention over the encoder output before applying its own feed-forward network and normalization. A final linear projection and softmax convert decoder states into next-token probabilities.

This is a blueprint rather than a fixed recipe. Encoder-only models such as BERT are built for understanding and representation learning; decoder-only models such as GPT generate text autoregressively; and encoder-decoder models such as T5 transform one sequence into another. Modern variants may replace sinusoidal positional encoding with RoPE and optimize attention with grouped-query attention or other efficiency improvements. The shared core remains repeated attention and feed-forward transformations connected by residual pathways.

How Self-Attention Works

For each token, learned projections produce a query, key, and value vector. The model compares each query with every key, scales the dot products by the key dimension, applies softmax, and uses the resulting weights to combine the value vectors. In shorthand, scaled dot-product attention is softmax(QK^T / sqrt(d_k))V. Multi-head attention runs several of these operations in parallel, allowing different heads to learn different relationships such as syntax, coreference, or local phrase structure.

This direct connection between positions gives a transformer a short path for relating distant tokens, but it has a cost: full self-attention performs O(n^2) pairwise interactions for a sequence of length n. Long-context systems therefore use optimized kernels, sparse or local attention, grouped-query attention, and other memory-saving techniques when the full quadratic pattern becomes too expensive.

Training and Inference

Training is highly parallel. A decoder-only model can evaluate every position in a training sequence at once while a causal mask prevents a token from seeing the future; encoder-only models commonly learn through masked-token objectives, while encoder-decoder models learn to transform one sequence into another. During generation, however, a decoder-only model produces tokens sequentially, so latency grows with the number of generated tokens. KV caching stores previously computed key and value projections, avoiding repeated work at each step. Batching, mixed-precision arithmetic, and memory-efficient attention kernels further improve throughput.

Transformers have become the dominant architecture in natural language processing and are increasingly used in computer vision, audio processing, and multimodal AI. Models like GPT, BERT, and their successors are all built on the transformer architecture, and they power today's most capable large language models.