Content deleted Content added
Tag: Reverted |
Tag: Reverted |
||
Line 42:
== Neural architecture search benchmarks ==
Neural architecture search often requires large computational resources, due to its expensive training and evaluation phases. This further leads to a large carbon footprint required for the evaluation of these methods. To overcome this limitation, NAS benchmarks<ref>Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K. and Hutter, F., 2019, May. Nas-bench-101: [[arxiv:1902.09635|Towards reproducible neural architecture search]]. In ''International Conference on Machine Learning'' (pp. 7105-7114). PMLR.</ref><ref>Zela, A., Siems, J. and Hutter, F., 2020. Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search. ''arXiv preprint [[arXiv:2001.10422]]''.</ref><ref>Dong, X. and Yang, Y., 2020. Nas-bench-201: Extending the scope of reproducible neural architecture search. ''arXiv preprint [[arXiv:2001.00326]]''.</ref><ref>Siems, J., Zimmer, L., Zela, A., Lukasik, J., Keuper, M. and Hutter, F., 2020. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. ''arXiv preprint [[arXiv:2008.09777]]''.</ref><ref>Hakim, T., 2022, September. NAAP-440 Dataset and Baseline for Neural Architecture Accuracy Prediction. ''arXiv preprint [[arXiv:2209.06626]]''.</ref> have been introduced, from which one can either query or predict the final performance of neural architectures in seconds. A NAS benchmark is defined as a dataset with a fixed train-test split, a search space, and a fixed training pipeline (hyperparameters). There are primarily two types of NAS benchmarks: a surrogate NAS benchmark and a tabular NAS benchmark. A surrogate benchmark uses a surrogate model (eg: a neural network) to predict the performance of an architecture from the search space. On the other hand a tabular benchmark queries the actual performance of an architecture trained upto convergence. Both of these benchmarks are queryable and can be used to efficiently simulate many NAS algorithms using only a CPU to query the benchmark instead of training an architecture from scratch.
==See also==
| |||