Universal Approximation Theorem

The Universal Approximation Theorem says that a feed-forward neural network with one hidden layer and a suitable non-polynomial activation can approximate any continuous function on a compact domain as closely as desired, provided it has enough hidden units. Cybenko's 1989 result established the idea for sigmoid-like activations; later work broadened the result to other activation families and architectures.

The theorem is about expressiveness, not performance. It does not tell us how many neurons are needed, whether gradient descent will find the useful weights, how much data training requires, or whether the learned function will generalize. A network can have the capacity to represent a target and still be too difficult to optimize or too data-hungry to deploy.

Its practical lesson is that representation is only the first bottleneck. Modern systems spend more effort on optimization, data quality, regularization, and efficient architecture design. Deep networks often beat very wide shallow ones because compositional structure can be expressed with far fewer parameters.