Neural Architecture Transfer 2

A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Search

Abstract

The advent of deep learning has had a significant impact on various sectors of modern society, with artificial neural networks becoming the leading models for tackling a wide range of challenges. The innovation of Neural Architecture Search (NAS) methods, which facilitate the automated creation of optimal neural networks, marks a significant step forward in this field. However, the large computational resources and time required for NAS processes are significant limitations. To address these challenges, Once-For-All (OFA) and its advanced version, Once-For-All-2 (OFAv2), were introduced to develop a single, comprehensive super-network capable of efficiently deriving specific sub-networks without the need for retraining, thereby maintaining stellar performance under varying constraints. Building on this, Neural Architecture Transfer (NAT) was developed to improve the efficiency of extracting such sub-networks from the overarching super-network. This study introduces Neural Architecture Transfer 2 (NAT2), an evolution of NAT that refines the multi-objective search mechanisms within dynamic super-networks to further improve the performance-complexity trade-off for the searched architectures. Leveraging the advances of OFAv2, NAT2 introduces significant qualitative improvements in the sub-networks that can be extracted by incorporating novel policies for network initialisation, pre-processing, and archive updates, as well as a fine-tuning based post-processing pipeline. The empirical evidence presented here highlights the effectiveness of NAT2 over its predecessor, particularly in the development of high-performance architectures with a reduced number of parameters and multiply-accumulate operations.