Neural Architecture Search

Neural Architecture Search (NAS) is the process of automatically searching for the best neural network architecture for a given task, rather than relying only on human intuition and manual experimentation. Instead of an engineer hand-designing every layer, connection pattern, kernel size, and width/depth tradeoff, a NAS system explores many candidate architectures and evaluates which ones perform best under a chosen objective.

A NAS system usually has three parts: a search space (the kinds of architectures it is allowed to consider), a search strategy (how it explores those candidates), and an evaluation method (how it judges whether one architecture is better than another). Search strategies have included reinforcement learning, evolutionary algorithms, gradient-based optimization, and more efficient one-shot or weight-sharing approaches.

The appeal of NAS is obvious: architecture design is one of the most expensive parts of deep learning research, and good architectures can produce large gains in accuracy, efficiency, or latency. NAS helped popularize the broader AutoML idea that not just model weights, but model structures themselves, can be optimized automatically.

Classic examples include NASNet, where Google used reinforcement learning to search for image-classification cells, and EfficientNet, whose scaling rules were derived from architecture-search ideas in combination with compound scaling. In practice, NAS has also been used for hardware-aware design, where the goal is not just maximum accuracy but the best tradeoff between quality, parameter count, memory use, and inference speed on a target device.

The main advantage of NAS is that it can discover architectures humans might not think to try, especially when optimizing for unusual hardware or multi-objective constraints. The main disadvantage is cost: early NAS methods were famously expensive, sometimes requiring enormous compute budgets just to search the design space. That expense is why later work focused heavily on making NAS cheaper and more practical through proxy tasks, shared weights, and differentiable search.

A useful way to think about NAS is that ordinary training asks, "What are the best weights for this architecture?" Neural Architecture Search asks the prior question: "What architecture should we be training in the first place?"