Neural architecture search: Difference between revisions

Revision as of 04:26, 17 June 2024

Revision as of 04:31, 31 July 2024

Line 39:

More recent works further combine this weight-sharing paradigm, with a continuous relaxation of the search space,<ref>{{cite arXiv |eprint=1812.00332 |last1=Cai |first1=Han |last2=Zhu |first2=Ligeng |last3=Han |first3=Song |title=ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware |date=2018 |class=cs.LG }}</ref><ref>{{cite arXiv |eprint=1910.04465 |last1=Dong |first1=Xuanyi |last2=Yang |first2=Yi |title=Searching for a Robust Neural Architecture in Four GPU Hours |date=2019 |class=cs.CV }}</ref><ref name="H. Liu, K. Simonyan 1806">{{cite arXiv |eprint=1806.09055 |last1=Liu |first1=Hanxiao |last2=Simonyan |first2=Karen |last3=Yang |first3=Yiming |title=DARTS: Differentiable Architecture Search |date=2018 |class=cs.LG }}</ref><ref>{{cite arXiv |eprint=1812.09926 |last1=Xie |first1=Sirui |last2=Zheng |first2=Hehui |last3=Liu |first3=Chunxiao |last4=Lin |first4=Liang |title=SNAS: Stochastic Neural Architecture Search |date=2018 |class=cs.LG }}</ref> which enables the use of gradient-based optimization methods. These approaches are generally referred to as differentiable NAS and have proven very efficient in exploring the search space of neural architectures. One of the most popular algorithms amongst the gradient-based methods for NAS is DARTS.<ref name="H. Liu, K. Simonyan 1806"/> However, DARTS faces problems such as performance collapse due to an inevitable aggregation of skip connections and poor generalization which were tackled by many future algorithms.<ref>{{cite arXiv |eprint=1911.12126 |last1=Chu |first1=Xiangxiang |last2=Zhou |first2=Tianbao |last3=Zhang |first3=Bo |last4=Li |first4=Jixiang |title=Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search |date=2019 |class=cs.LG }}</ref><ref name="Arber Zela 1909">{{cite arXiv |eprint=1909.09656 |last1=Zela |first1=Arber |last2=Elsken |first2=Thomas |last3=Saikia |first3=Tonmoy |last4=Marrakchi |first4=Yassine |last5=Brox |first5=Thomas |last6=Hutter |first6=Frank |title=Understanding and Robustifying Differentiable Architecture Search |date=2019 |class=cs.LG }}</ref><ref name="Xiangning Chen 2002">{{cite arXiv |eprint=2002.05283 |last1=Chen |first1=Xiangning |last2=Hsieh |first2=Cho-Jui |title=Stabilizing Differentiable Architecture Search via Perturbation-based Regularization |date=2020 |class=cs.LG }}</ref><ref>{{cite arXiv |eprint=1907.05737 |last1=Xu |first1=Yuhui |last2=Xie |first2=Lingxi |last3=Zhang |first3=Xiaopeng |last4=Chen |first4=Xin |last5=Qi |first5=Guo-Jun |last6=Tian |first6=Qi |last7=Xiong |first7=Hongkai |title=PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search |date=2019 |class=cs.CV }}</ref> Methods like <ref name="Arber Zela 1909"/><ref name="Xiangning Chen 2002"/> aim at robustifying DARTS and making the validation accuracy landscape smoother by introducing a Hessian norm based regularisation and random smoothing/adversarial attack respectively. The cause of performance degradation is later analyzed from the architecture selection aspect.<ref>{{cite arXiv |eprint=2108.04392 |last1=Wang |first1=Ruochen |last2=Cheng |first2=Minhao |last3=Chen |first3=Xiangning |last4=Tang |first4=Xiaocheng |last5=Hsieh |first5=Cho-Jui |title=Rethinking Architecture Selection in Differentiable NAS |date=2021 |class=cs.LG }}</ref>

Differentiable NAS has shown to produce competitive results using a fraction of the search-time required by RL-based search methods. For example, FBNet (which is short for Facebook Berkeley Network) demonstrated that supernetwork-based search produces networks that outperform the speed-accuracy tradeoff curve of mNASNet and MobileNetV2 on the ImageNet image-classification dataset. FBNet accomplishes this using over 400x ''less'' search time than was used for mNASNet.<ref name="FBNet">{{cite arXiv|eprint=1812.03443|last1=Wu|first1=Bichen|title=FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search|last2=Dai|first2=Xiaoliang|last3=Zhang|first3=Peizhao|last4=Wang|first4=Yanghan|last5=Sun|first5=Fei|last6=Wu|first6=Yiming|last7=Tian|first7=Yuandong|last8=Vajda|first8=Peter|last9=Jia|first9=Yangqing|last10=Keutzer|first10=Kurt|class=cs.CV|date=24 May 2019}}</ref><ref name="MobileNetV2">{{cite arXiv|eprint=1801.04381|last1=Sandler|first1=Mark|title=MobileNetV2: Inverted Residuals and Linear Bottlenecks|last2=Howard|first2=Andrew|last3=Zhu|first3=Menglong|last4=Zhmoginov|first4=Andrey|last5=Chen|first5=Liang-Chieh|class=cs.CV|year=2018}}</ref><ref>{{Cite web|url=httphttps://sitessite.ieee.org/scv-cas/files/2019/05/2019-05-22-ieee-co-design-trim.pdf|title=Co-Design of DNNs and NN Accelerators|last=Keutzer|first=Kurt|date=2019-05-22|website=IEEE|url-status=|archive-url=|archive-date=|access-date=2019-09-26}}</ref> Further, SqueezeNAS demonstrated that supernetwork-based NAS produces neural networks that outperform the speed-accuracy tradeoff curve of MobileNetV3 on the Cityscapes semantic segmentation dataset, and SqueezeNAS uses over 100x less search time than was used in the MobileNetV3 authors' RL-based search.<ref name="SqueezeNAS">{{cite arXiv|eprint=1908.01748|last1=Shaw|first1=Albert|title=SqueezeNAS: Fast neural architecture search for faster semantic segmentation|last2=Hunter|first2=Daniel|last3=Iandola|first3=Forrest|last4=Sidhu|first4=Sammy|class=cs.CV|year=2019}}</ref><ref>{{Cite news|url=https://www.eetimes.com/document.asp?doc_id=1335063|title=Does Your AI Chip Have Its Own DNN?|last=Yoshida|first=Junko|date=2019-08-25|work=EE Times|access-date=2019-09-26}}</ref>

== Neural architecture search benchmarks ==