Weight Initialization
Models & ArchitecturesThe rule used to choose a neural network’s starting weights so activations and gradients remain in a trainable range.
It is launch calibration for a signal: if every layer amplifies or muffles it, the network loses control before learning begins.
Weight initialization chooses the parameter values a network receives before its first update. Set every weight to zero and neurons remain symmetric, learning the same feature. Start with values that are too large and activations or gradients can explode; start too small and they can disappear.
The usual goal is variance preservation. Xavier initialization uses Var(w) = 2/(n_in + n_out) to balance signal flow for activations such as tanh. He initialization uses Var(w) = 2/n_in and accounts for ReLU dropping roughly half of its inputs. Biases are often initialized to zero or to a small task-specific value.
Initialization is not an isolated trick. It interacts with the activation function, normalization scheme, architecture, optimizer, and precision. Modern frameworks provide sensible defaults, but custom layers, unusual residual paths, and very deep networks still benefit from checking activation statistics and gradient norms at the start of training.
Last updated: February 22, 2026