Neural architecture search: Difference between revisions

Revision as of 01:04, 4 February 2024

Latest revision as of 00:29, 27 August 2025

(8 intermediate revisions by 7 users not shown)

Line 1:

{{Short description|Machine learning-powered structure design}}

{{Machine learning bar}}

'''Neural architecture search''' ('''NAS''')<ref name="survey">{{Cite journal|url=http://jmlr.org/papers/v20/18-598.html|title=Neural Architecture Search: A Survey|first1=Thomas|last1=Elsken|first2=Jan Hendrik|last2=Metzen|first3=Frank|last3=Hutter|date=August 8, 2019|journal=Journal of Machine Learning Research|volume=20|issue=55|pages=1–21|arxiv=1808.05377}}</ref><ref name="survey2">{{cite arXiv|last1=Wistuba|first1=Martin|last2=Rawat|first2=Ambrish|last3=Pedapati|first3=Tejaswini|date=2019-05-04|title=A Survey on Neural Architecture Search|eprint=1905.01392|class=cs.LG}}</ref> is a technique for automating the design of [[artificial neural network]]s (ANN), a widely used model in the field of [[machine learning]]. NAS has been used to design networks that are on par with or outperform hand-designed architectures.<ref name="Zoph 2016" /><ref name="Zoph 2017">{{cite arXiv|last1=Zoph|first1=Barret|last2=Vasudevan|first2=Vijay|last3=Shlens|first3=Jonathon|last4=Le|first4=Quoc V.|date=2017-07-21|title=Learning Transferable Architectures for Scalable Image Recognition|eprint=1707.07012|class=cs.CV}}</ref> Methods for NAS can be categorized according to the search space, search strategy and performance estimation strategy used:<ref name="survey" />

* The ''search space'' defines the type(s) of ANN that can be designed and optimized.

Line 7:

* The ''performance estimation strategy'' evaluates the performance of a possible ANN from its design (without constructing and training it).

NAS is closely related to [[hyperparameter optimization]]<ref>Matthias Feurer and Frank Hutter. [https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5_1.pdf Hyperparameter optimization]. In: ''AutoML: Methods, Systems, Challenges'', pages 3–38.</ref> and [[meta-learning (computer science)|meta-learning]]<ref>{{Cite book|chapter-url=https://link.springer.com/chapter/10.1007/978-3-030-05318-5_2|doi = 10.1007/978-3-030-05318-5_2|chapter = Meta-Learning|title = Automated Machine Learning|series = The Springer Series on Challenges in Machine Learning|year = 2019|last1 = Vanschoren|first1 = Joaquin|pages = 35–61|isbn = 978-3-030-05317-8|s2cid = 239362577}}</ref> and is a subfield of [[automated machine learning]] (AutoML).<ref>{{Cite journal |last1=Salehin |first1=Imrus |last2=Islam |first2=Md. Shamiul |last3=Saha |first3=Pritom |last4=Noman |first4=S. M. |last5=Tuni |first5=Azra |last6=Hasan |first6=Md. Mehedi |last7=Baten |first7=Md. Abu |date=2024-01-01 |title=AutoML: A systematic review on automated machine learning with neural architecture search |journal=Journal of Information and Intelligence |volume=2 |issue=1 |pages=52–81 |doi=10.1016/j.jiixd.2023.10.002 |issn=2949-7159|doi-access=free }}</ref>

==Reinforcement learning==

Line 17:

== Evolution ==

An alternative approach to NAS is based on [[evolutionary algorithm]]s, which has been employed by several groups.<ref>{{cite arXiv|last1=Real|first1=Esteban|last2=Moore|first2=Sherry|last3=Selle|first3=Andrew|last4=Saxena|first4=Saurabh|last5=Suematsu|first5=Yutaka Leon|last6=Tan|first6=Jie|last7=Le|first7=Quoc|last8=Kurakin|first8=Alex|date=2017-03-03|title=Large-Scale Evolution of Image Classifiers|eprint=1703.01041|class=cs.NE}}</ref><ref>{{Cite arXiv|last1=Suganuma|first1=Masanori|last2=Shirakawa|first2=Shinichi|last3=Nagao|first3=Tomoharu|date=2017-04-03|title=A Genetic Programming Approach to Designing Convolutional Neural Network Architectures|class=cs.NE|eprint=1704.00764v2|language=en}}</ref><ref name=":0">{{Cite arXiv|last1=Liu|first1=Hanxiao|last2=Simonyan|first2=Karen|last3=Vinyals|first3=Oriol|last4=Fernando|first4=Chrisantha|last5=Kavukcuoglu|first5=Koray|date=2017-11-01|title=Hierarchical Representations for Efficient Architecture Search|class=cs.LG|eprint=1711.00436v2|language=en}}</ref><ref name="Real 2018">{{cite arXiv|last1=Real|first1=Esteban|last2=Aggarwal|first2=Alok|last3=Huang|first3=Yanping|last4=Le|first4=Quoc V.|date=2018-02-05|title=Regularized Evolution for Image Classifier Architecture Search|eprint=1802.01548|class=cs.NE}}</ref><ref>{{cite arXiv|last1=Miikkulainen|first1=Risto|last2=Liang|first2=Jason|last3=Meyerson|first3=Elliot|last4=Rawal|first4=Aditya|last5=Fink|first5=Dan|last6=Francon|first6=Olivier|last7=Raju|first7=Bala|last8=Shahrzad|first8=Hormoz|last9=Navruzyan|first9=Arshak|last10=Duffy|first10=Nigel|last11=Hodjat|first11=Babak|date=2017-03-04|title=Evolving Deep Neural Networks|class=cs.NE|eprint=1703.00548}}</ref><ref>{{Cite book|last1=Xie|first1=Lingxi|last2=Yuille|first2=Alan|title=2017 IEEE International Conference on Computer Vision (ICCV) |chapter=Genetic CNN |chapter-url=https://ieeexplore.ieee.org/document/8237416|year=2017|pages=1388–1397|doi=10.1109/ICCV.2017.154|arxiv=1703.01513|isbn=978-1-5386-1032-9|s2cid=206770867}}</ref><ref name="Elsken 2018" /> An Evolutionary Algorithm for Neural Architecture Search generally performs the following procedure.<ref name="liu2021survey">{{cite journal|last1=Liu|first1=Yuqiao|last2=Sun|first2=Yanan|last3=Xue|first3=Bing|last4=Zhang|first4=Mengjie|last5=Yen|first5=Gary G|last6=Tan|first6=Kay Chen|title=A Survey on Evolutionary Neural Architecture Search|journal=IEEE Transactions on Neural Networks and Learning Systems|year=2021|volume= 34|issue=2 |pages=1–21|doi=10.1109/TNNLS.2021.3100554|pmid=34357870|arxiv=2008.10937|s2cid=221293236}}</ref> First a pool consisting of different candidate architectures along with their validation scores (fitness) is initialised. At each step the architectures in the candidate pool are mutated (e.g.: 3x3 convolution instead of a 5x5 convolution). Next the new architectures are trained from scratch for a few epochs and their validation scores are obtained. This is followed by replacing the lowest scoring architectures in the candidate pool with the better, newer architectures. This procedure is repeated multiple times and thus the candidate pool is refined over time. Mutations in the context of evolving ANNs are operations such as adding or removing a layer, which include changing the type of a layer (e.g., from convolution to pooling), changing the hyperparameters of a layer, or changing the training hyperparameters. On [[CIFAR-10]] and [[ImageNet]], evolution and RL performed comparably, while both slightly outperformed [[random search]].<ref name="Real 2018" /><ref name=":0" />

== Bayesian optimization ==

Line 26:

== Multi-objective search ==

While most approaches solely focus on finding architecture with maximal predictive performance, for most practical applications other objectives are relevant, such as memory consumption, model size or inference time (i.e., the time required to obtain a prediction). Because of that, researchers created a [[Multi-objective optimization|multi-objective]] search.<ref name="Elsken 2018">{{cite arXiv|last1=Elsken|first1=Thomas|last2=Metzen|first2=Jan Hendrik|last3=Hutter|first3=Frank|date=2018-04-24|title=Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution|eprint=1804.09081|class=stat.ML}}</ref><ref name="Zhou 2018">{{cite web|url=https://www.sysml.cc/doc/2018/94.pdf|title=Neural Architect: A Multi-objective Neural Architecture Search with Performance Prediction|last1=Zhou|first1=Yanqi|last2=Diamos|first2=Gregory|date=|website=|publisher=Baidu|access-date=2019-09-27|archive-date=2019-09-27|archive-url=https://web.archive.org/web/20190927090457/https://www.sysml.cc/doc/2018/94.pdf|url-status=dead}}</ref>

LEMONADE<ref name="Elsken 2018" /> is an evolutionary algorithm that adopted [[Lamarckism]] to efficiently optimize multiple objectives. In every generation, child networks are generated to improve the [[Pareto efficiency#Pareto frontier|Pareto frontier]] with respect to the current population of ANNs.

Line 39:

More recent works further combine this weight-sharing paradigm, with a continuous relaxation of the search space,<ref>{{cite arXiv |eprint=1812.00332 |last1=Cai |first1=Han |last2=Zhu |first2=Ligeng |last3=Han |first3=Song |title=ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware |date=2018 |class=cs.LG }}</ref><ref>{{cite arXiv |eprint=1910.04465 |last1=Dong |first1=Xuanyi |last2=Yang |first2=Yi |title=Searching for a Robust Neural Architecture in Four GPU Hours |date=2019 |class=cs.CV }}</ref><ref name="H. Liu, K. Simonyan 1806">{{cite arXiv |eprint=1806.09055 |last1=Liu |first1=Hanxiao |last2=Simonyan |first2=Karen |last3=Yang |first3=Yiming |title=DARTS: Differentiable Architecture Search |date=2018 |class=cs.LG }}</ref><ref>{{cite arXiv |eprint=1812.09926 |last1=Xie |first1=Sirui |last2=Zheng |first2=Hehui |last3=Liu |first3=Chunxiao |last4=Lin |first4=Liang |title=SNAS: Stochastic Neural Architecture Search |date=2018 |class=cs.LG }}</ref> which enables the use of gradient-based optimization methods. These approaches are generally referred to as differentiable NAS and have proven very efficient in exploring the search space of neural architectures. One of the most popular algorithms amongst the gradient-based methods for NAS is DARTS.<ref name="H. Liu, K. Simonyan 1806"/> However, DARTS faces problems such as performance collapse due to an inevitable aggregation of skip connections and poor generalization which were tackled by many future algorithms.<ref>{{cite arXiv |eprint=1911.12126 |last1=Chu |first1=Xiangxiang |last2=Zhou |first2=Tianbao |last3=Zhang |first3=Bo |last4=Li |first4=Jixiang |title=Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search |date=2019 |class=cs.LG }}</ref><ref name="Arber Zela 1909">{{cite arXiv |eprint=1909.09656 |last1=Zela |first1=Arber |last2=Elsken |first2=Thomas |last3=Saikia |first3=Tonmoy |last4=Marrakchi |first4=Yassine |last5=Brox |first5=Thomas |last6=Hutter |first6=Frank |title=Understanding and Robustifying Differentiable Architecture Search |date=2019 |class=cs.LG }}</ref><ref name="Xiangning Chen 2002">{{cite arXiv |eprint=2002.05283 |last1=Chen |first1=Xiangning |last2=Hsieh |first2=Cho-Jui |title=Stabilizing Differentiable Architecture Search via Perturbation-based Regularization |date=2020 |class=cs.LG }}</ref><ref>{{cite arXiv |eprint=1907.05737 |last1=Xu |first1=Yuhui |last2=Xie |first2=Lingxi |last3=Zhang |first3=Xiaopeng |last4=Chen |first4=Xin |last5=Qi |first5=Guo-Jun |last6=Tian |first6=Qi |last7=Xiong |first7=Hongkai |title=PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search |date=2019 |class=cs.CV }}</ref> Methods like <ref name="Arber Zela 1909"/><ref name="Xiangning Chen 2002"/> aim at robustifying DARTS and making the validation accuracy landscape smoother by introducing a Hessian norm based regularisation and random smoothing/adversarial attack respectively. The cause of performance degradation is later analyzed from the architecture selection aspect.<ref>{{cite arXiv |eprint=2108.04392 |last1=Wang |first1=Ruochen |last2=Cheng |first2=Minhao |last3=Chen |first3=Xiangning |last4=Tang |first4=Xiaocheng |last5=Hsieh |first5=Cho-Jui |title=Rethinking Architecture Selection in Differentiable NAS |date=2021 |class=cs.LG }}</ref>

Differentiable NAS has shown to produce competitive results using a fraction of the search-time required by RL-based search methods. For example, FBNet (which is short for Facebook Berkeley Network) demonstrated that supernetwork-based search produces networks that outperform the speed-accuracy tradeoff curve of mNASNet and MobileNetV2 on the ImageNet image-classification dataset. FBNet accomplishes this using over 400x ''less'' search time than was used for mNASNet.<ref name="FBNet">{{cite arXiv|eprint=1812.03443|last1=Wu|first1=Bichen|title=FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search|last2=Dai|first2=Xiaoliang|last3=Zhang|first3=Peizhao|last4=Wang|first4=Yanghan|last5=Sun|first5=Fei|last6=Wu|first6=Yiming|last7=Tian|first7=Yuandong|last8=Vajda|first8=Peter|last9=Jia|first9=Yangqing|last10=Keutzer|first10=Kurt|class=cs.CV|date=24 May 2019}}</ref><ref name="MobileNetV2">{{cite arXiv|eprint=1801.04381|last1=Sandler|first1=Mark|title=MobileNetV2: Inverted Residuals and Linear Bottlenecks|last2=Howard|first2=Andrew|last3=Zhu|first3=Menglong|last4=Zhmoginov|first4=Andrey|last5=Chen|first5=Liang-Chieh|class=cs.CV|year=2018}}</ref><ref>{{Cite web|url=httphttps://sitessite.ieee.org/scv-cas/files/2019/05/2019-05-22-ieee-co-design-trim.pdf|title=Co-Design of DNNs and NN Accelerators|last=Keutzer|first=Kurt|date=2019-05-22|website=IEEE|url-status=|archive-url=|archive-date=|access-date=2019-09-26}}</ref> Further, SqueezeNAS demonstrated that supernetwork-based NAS produces neural networks that outperform the speed-accuracy tradeoff curve of MobileNetV3 on the Cityscapes semantic segmentation dataset, and SqueezeNAS uses over 100x less search time than was used in the MobileNetV3 authors' RL-based search.<ref name="SqueezeNAS">{{cite arXiv|eprint=1908.01748|last1=Shaw|first1=Albert|title=SqueezeNAS: Fast neural architecture search for faster semantic segmentation|last2=Hunter|first2=Daniel|last3=Iandola|first3=Forrest|last4=Sidhu|first4=Sammy|class=cs.CV|year=2019}}</ref><ref>{{Cite news|url=https://www.eetimes.com/document.asp?doc_id=1335063|title=Does Your AI Chip Have Its Own DNN?|last=Yoshida|first=Junko|date=2019-08-25|work=EE Times|access-date=2019-09-26}}</ref>

== Neural architecture search benchmarks ==

Line 46:

==See also==

*[[Neural Network Intelligence]]

*[[automated machine learning|Automated Machine Learning]]

*[[hyperparameter optimization|Hyperparameter Optimization]]

== Further reading ==

Line 59 ⟶ 61:

{{Differentiable computing}}

[[Category:Artificial intelligence engineering]]

[[Category:Artificial intelligence]]