Vanishing Gradients
Models & ArchitecturesA backpropagation failure mode in which gradient signals shrink across layers or time steps, leaving early parameters with almost no useful update.
Backpropagation is a courier route through the network; if every layer trims the message, the first layer receives a faint receipt instead of an instruction.
Vanishing gradients appear when the chain rule repeatedly shrinks the learning signal as it moves backward through a network. If each layer contributes a factor of 0.9, then after 50 layers the signal is 0.9^50, or about 0.005. The earliest layers receive almost no update, even when the final prediction is wrong.
Sigmoid and tanh activations made the problem especially severe because their derivatives approach zero in saturated regions. Recurrent networks face the same obstacle across time: information from an early token may need to survive dozens or thousands of repeated transformations before it can influence a later output.
Modern architectures reduce the risk rather than making it disappear. ReLU and related activations preserve stronger gradients on their active side; Xavier and He initialization keep signal scales controlled; residual connections create shorter paths for backpropagation; and LSTM or GRU gates regulate memory in recurrent systems. Batch normalization and layer normalization can help keep activations in a trainable range.
Last updated: February 22, 2026