Content deleted Content added
No edit summary |
No edit summary Tag: Disambiguation links added |
||
Line 136:
A class of early stopping-based hyperparameter optimization algorithms is purpose built for large search spaces of continuous and discrete hyperparameters, particularly when the computational cost to evaluate the performance of a set of hyperparameters is high. Irace implements the iterated racing algorithm, that focuses the search around the most promising configurations, using statistical tests to discard the ones that perform poorly.<ref name="irace">{{cite journal |last1=López-Ibáñez |first1=Manuel |last2=Dubois-Lacoste |first2=Jérémie |last3=Pérez Cáceres |first3=Leslie |last4=Stützle |first4=Thomas |last5=Birattari |first5=Mauro |date=2016 |title=The irace package: Iterated Racing for Automatic Algorithm Configuration |journal=Operations Research Perspective |volume=3 |issue=3 |pages=43–58 |doi=10.1016/j.orp.2016.09.002|doi-access=free }}</ref><ref name="race">{{cite journal |last1=Birattari |first1=Mauro |last2=Stützle |first2=Thomas |last3=Paquete |first3=Luis |last4=Varrentrapp |first4=Klaus |date=2002 |title=A Racing Algorithm for Configuring Metaheuristics |journal=Gecco 2002 |pages=11–18}}</ref>
Another early stopping hyperparameter optimization algorithm is successive halving (SHA),<ref>{{cite arXiv|last1=Jamieson|first1=Kevin|last2=Talwalkar|first2=Ameet|date=2015-02-27|title=Non-stochastic Best Arm Identification and Hyperparameter Optimization|eprint=1502.07943|class=cs.LG}}</ref> which begins as a random search but periodically prunes low-performing models, thereby focusing computational resources on more promising models. Asynchronous successive halving (ASHA)<ref>{{cite arXiv|last1=Li|first1=Liam|last2=Jamieson|first2=Kevin|last3=Rostamizadeh|first3=Afshin|last4=Gonina|first4=Ekaterina|last5=Hardt|first5=Moritz|last6=Recht|first6=Benjamin|last7=Talwalkar|first7=Ameet|date=2020-03-16|title=A System for Massively Parallel Hyperparameter Tuning|class=cs.LG|eprint=1810.05934v5}}</ref> further improves upon SHA's resource utilization profile by removing the need to synchronously evaluate and prune low-performing models. Hyperband<ref>{{cite journal|last1=Li|first1=Lisha|last2=Jamieson|first2=Kevin|last3=DeSalvo|first3=Giulia|last4=Rostamizadeh|first4=Afshin|last5=Talwalkar|first5=Ameet|date=2020-03-16|title=Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization|journal=Journal of Machine Learning Research|volume=18|pages=1–52|arxiv=1603.06560}}</ref> is a higher level early stopping-based algorithm that invokes SHA or ASHA multiple times with varying levels of pruning aggressiveness, in order to be more widely applicable and with fewer required inputs.
=== Others ===
[[Radial basis function|RBF]]<ref name=abs1705.08520>{{cite arXiv |eprint=1705.08520|last1=Diaz|first1=Gonzalo|title=An effective algorithm for hyperparameter optimization of neural networks|last2=Fokoue|first2=Achille|last3=Nannicini|first3=Giacomo|last4=Samulowitz|first4=Horst|class=cs.AI|year=2017}}</ref> and [[spectral method|spectral]]<ref name=abs1706.00764>{{cite arXiv |eprint=1706.00764|last1=Hazan|first1=Elad|title=Hyperparameter Optimization: A Spectral Approach|last2=Klivans|first2=Adam|last3=Yuan|first3=Yang|class=cs.LG|year=2017}}</ref> approaches have also been developed.
== Issues of hyperparameter optimization ==
When hyperparameter optimization is done, the set of hyperparameters are often fitted on a training set and selected based on the generalization performance, or score, of a validation set. However, this procedure is at risk of overfitting the hyperparameters to the validation set. Therefore, the generalization performance score of the validation set (which can be several sets in the case of a cross-validation procedure) cannot be used to simultanesouly estimate the generalization performance of the final model. In order to do so, the generalization performance has to be evaluated on a set independent (which has no intersection) of the set (or sets) used for the optimization of the hyperparameters, otherwise the performance might give a value which is too optimistic (too large). This can be done on a second test set, or through an outer [[cross-validation]] procedure called nested cross-validation, which allows an unbiased estimation of the generalization performance of the model, taking into account the bias due to the hyperparameter optimization.
== See also ==
| |||