From Coarse to Fine: A Multi-Stage Framework for Neural Architecture Search [Research Frontier]

Abstract

Neural Architecture Search (NAS) offers a promising pathway to automate the design of deep neural networks, yet it faces a significant challenge in balancing computational efficiency with search effectiveness in a huge search space. Coarse-to-fine (C2F) NAS methods mitigate this by narrowing the architecture search to promising subspaces. However, by focusing only on a single region, they risk overlooking globally optimal architectures due to the multi-modal nature of the search space. This paper highlights the limitation of existing C2F approaches and motivates the need for more sophisticated search strategies capable of efficiently exploring multiple promising regions to achieve a better performance-efficiency trade-off. To achieve this goal, we propose a multi-stage NAS (MstageNAS) framework that progressively constructs multiple high-quality subspaces and implements an efficient exploration within them. MstageNAS initiates with a coarse search to identify promising architectures within the whole search space. Subsequently, the individual subspace is constructed around each promising architecture. To ensure the quality of this subspace, an architecture explanation method is devised to identify the core sub-structure of the promising architecture and use it to form the basis of the individual subspace. Finally, a Monte Carlo-based search strategy is developed to facilitate architecture search within these subspaces, with the goal of striking a good balance between exploration and exploitation. We evaluate the proposed MstageNAS framework across search spaces of various types and tasks. Extensive experiments demonstrate that MstageNAS can outperform state-of-the-art NAS methods or achieve comparable performance but with about 2× less search cost.