Hyperparameter optimization: Difference between revisions

Revision as of 13:15, 19 January 2018

Latest revision as of 18:34, 13 August 2025

(209 intermediate revisions by 89 users not shown)

Line 1:

{{Short description|Process of finding the optimal set of variables

In [[machine learning]], '''hyperparameter optimization''' or tuning is the problem of choosing a set of optimal [[Hyperparameter (machine learning)|hyperparameters]] for a learning algorithm.

for a machine learning algorithm}}

In [[machine learning]], '''hyperparameter optimization'''<ref>Matthias Feurer and Frank Hutter. [https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5_1.pdf Hyperparameter optimization]. In: ''AutoML: Methods, Systems, Challenges'', pages 3–38.</ref> or tuning is the problem of choosing a set of optimal [[Hyperparameter (machine learning)|hyperparameters]] for a learning algorithm. A hyperparameter is a [[parameter]] whose value is used to control the learning process, which must be configured before the process starts.<ref>{{cite journal |last1=Yang|first1=Li|title=On hyperparameter optimization of machine learning algorithms: Theory and practice|journal=Neurocomputing|year=2020|volume=415|pages=295–316|doi=10.1016/j.neucom.2020.07.061|arxiv=2007.15745 }}</ref><ref>{{cite arXiv |vauthors=Franceschi L, Donini M, Perrone V, Klein A, Archambeau C, Seeger M, Pontil M, Frasconi P |title=Hyperparameter Optimization in Machine Learning |year=2024 |class=stat.ML |eprint=2410.22854 }}</ref>

The same kind of machine learning model can require different constraints, weights or learning rates to generalize different data patterns. These measures are called hyperparameters, and have to be tuned so that the model can optimally solve the machine learning problem. Hyperparameter optimization findsdetermines athe tupleset of hyperparameters that yields an optimal model which minimizes a predefined [[loss function]] on a given independent [[data set]].<ref name=abs1502.02127>{{cite arxivarXiv |eprint=1502.02127|last1=Claesen|first1=Marc|title=Hyperparameter Search in Machine Learning|author2=Bart De Moor|class=cs.LG|year=2015}}</ref> The objective function takes a tupleset of hyperparameters and returns the associated loss.<ref name=abs1502.02127/> [[Cross-validation (statistics)|Cross-validation]] is often used to estimate this generalization performance, and therefore choose the set of values for hyperparameters that maximize it.<ref name="bergstra">{{cite journal|last1=Bergstra|first1=James|last2=Bengio|first2=Yoshua|year=2012|title=Random Search for Hyper-Parameter Optimization|url=http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf|journal=Journal of Machine Learning Research|volume=13|pages=281–305}}</ref>

== Approaches ==

[[File:Hyperparameter Optimization using Grid Search.svg|thumb|Grid search across different values of two hyperparameters. For each hyperparameter, 10 different values are considered, so a total of 100 different combinations are evaluated and compared. Blue contours indicate regions with strong results, whereas red ones show regions with poor results.]]

=== Grid search ===

The traditional waymethod of performingfor hyperparameter optimization has been ''grid search'', or a ''parameter sweep'', which is simply an [[Brute-force search|exhaustive searching]] through a manually specified subset of the hyperparameter space of a learning algorithm. A grid search algorithm must be guided by some performance metric, typically measured by [[Cross-validation (statistics)|cross-validation]] on the training set<ref>Chin-Wei Hsu, Chih-Chung Chang and Chih-Jen Lin (2010). [http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf A practical guide to support vector classification]. Technical Report, [[National Taiwan University]].</ref>

or evaluation on a heldhold-out validation set.<ref>{{cite journal

| vauthors = Chicco D

| title = Ten quick tips for machine learning in computational biology

Line 13 ⟶ 17:

| volume = 10

| issue = 35

| pages = 1–1735

| date = December 2017

| pmid = 29234465

| doi = 10.1186/s13040-017-0155-3

| pmc= 5721660}}</ref>

| doi-access = free

}}</ref>

Since the parameter space of a machine learner may include real-valued or unbounded value spaces for certain parameters, manually set bounds and discretization may be necessary before applying grid search.

Line 28 ⟶ 34:

Grid search then trains an SVM with each pair (''C'', γ) in the [[Cartesian product]] of these two sets and evaluates their performance on a held-out validation set (or by internal cross-validation on the training set, in which case multiple SVMs are trained per pair). Finally, the grid search algorithm outputs the settings that achieved the highest score in the validation procedure.

Grid search suffers from the [[curse of dimensionality]], but is often [[embarrassingly parallel]] because typically the hyperparameter settings it evaluates are typically independent of each other.<ref name="bergstra"/>

[[File:Hyperparameter Optimization using Random Search.svg|thumb|Random search across different combinations of values for two hyperparameters. In this example, 100 different random choices are evaluated. The green bars show that more individual values for each hyperparameter are considered compared to a grid search.]]

=== Random search ===

Random Search replaces the exhaustive enumeration of all combinations by selecting them randomly. This can be simply applied to the discrete setting described above, but also generalizes to continuous and mixed spaces. A benefit over grid search is that random search can explore many more values than grid search could for continuous hyperparameters. It can outperform Grid search, especially when only a small number of hyperparameters affects the final performance of the machine learning algorithm.<ref name="bergstra" /> In this case, the optimization problem is said to have a low intrinsic dimensionality.<ref>{{Cite journal|last1=Ziyu|first1=Wang|last2=Frank|first2=Hutter|last3=Masrour|first3=Zoghi|last4=David|first4=Matheson|last5=Nando|first5=de Feitas|date=2016|title=Bayesian Optimization in a Billion Dimensions via Random Embeddings|journal=Journal of Artificial Intelligence Research|language=en|volume=55|pages=361–387|doi=10.1613/jair.4806|arxiv=1301.1942|s2cid=279236}}</ref> Random Search is also [[embarrassingly parallel]], and additionally allows the inclusion of prior knowledge by specifying the distribution from which to sample. Despite its simplicity, random search remains one of the important base-lines against which to compare the performance of new hyperparameter optimization methods.

{{main article|Random search}}

[[File:Hyperparameter Optimization using Tree-Structured Parzen Estimators.svg|thumb|Methods such as Bayesian optimization smartly explore the space of potential choices of hyperparameters by deciding which combination to explore next based on previous observations.]]

Since grid searching is an exhaustive and therefore potentially expensive method, several alternatives have been proposed. In particular, a randomized search that simply samples parameter settings a fixed number of times has been found to be more effective in high-dimensional spaces than exhaustive search. This is because oftentimes, it turns out some hyperparameters do not significantly affect the loss. Therefore, having randomly dispersed data gives more "textured" data than an exhaustive search over parameters that ultimately do not affect the loss.<ref name="bergstra">{{cite journal

| title = Random Search for Hyper-Parameter Optimization

| first1 = James

| last1 = Bergstra

| first2 = Yoshua

| last2 = Bengio

| journal = J. Machine Learning Research

| volume = 13

| year = 2012

| pages = 281–305

| url = http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf

}}</ref>

=== Bayesian optimization ===

{{main article|Bayesian optimization}}

Bayesian optimization is a methodology for the global optimization ofmethod for noisy black-box functions. Applied to hyperparameter optimization, Bayesian optimization consists of developingbuilds a statisticalprobabilistic model of the function mapping from hyperparameter values to the objective evaluated on a validation set. By Intuitively,iteratively theevaluating methodologya assumespromising thathyperparameter thereconfiguration isbased someon smooththe butcurrent noisymodel, functionand thatthen actsupdating as a mapping from hyperparameters to the objective. Init, Bayesian optimization, one aims to gather observations in such a manner as to evaluate the machine learning model the least number of times while revealing as much information as possible about this function and, in particular, the ___location of the optimum. It Bayesian optimization relies on assuming a very general prior over functions which when combined with observed hyperparameter values and corresponding outputs yields a distribution over functions. The methodology proceeds by iteratively picking hyperparameterstries to observe (experiments to run) in a manner that trades offbalance exploration (hyperparameters for which the outcome is most uncertain) and exploitation (hyperparameters whichexpected are expectedclose to havethe a good outcomeoptimum). In practice, Bayesian optimization has been shown<ref name="hutter">{{Citation

| lastlast1 = Hutter

| firstfirst1 = Frank

| last2 = Hoos

| first2 = Holger

| last3 = Leyton-Brown

| first3 = Kevin

| titlechapter = Sequential modelModel-basedBased optimizationOptimization for generalGeneral algorithmAlgorithm configurationConfiguration

| journaltitle = Learning and Intelligent Optimization

| volume = 6683

| pages = 507–523

| year = 2011

| url = http://www.cs.ubc.ca/labs/beta/Projects/SMAC/papers/11-LION5-SMAC.pdf }}</ref><ref| doi name="bergstra11">{{Citation 10.1007/978-3-642-25566-3_40

| citeseerx = 10.1.1.307.8813

| last = Bergstra

| series = Lecture Notes in Computer Science

| first = James

| isbn = 978-3-642-25565-6

| s2cid = 6944647

}}</ref><ref name="bergstra11">{{Citation

| last1 = Bergstra

| first1 = James

| last2 = Bardenet

| first2 = Remi

Line 72 ⟶ 76:

| year = 2011

| url = http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf }}</ref><ref name="snoek">{{cite journal

| lastlast1 = Snoek

| firstfirst1 = Jasper

| last2 = Larochelle

| first2 = Hugo

Line 86 ⟶ 90:

| bibcode = 2012arXiv1206.2944S

| arxiv = 1206.2944

| class = stat.ML

}}</ref><ref name="thornton">{{cite journal

| lastlast1 = Thornton

| firstfirst1 = Chris

| last2 = Hutter

| first2 = Frank

Line 104 ⟶ 107:

| bibcode = 2012arXiv1208.3719T

| arxiv = 1208.3719

}}</ref><ref name="krnc">{{Citation

| class = cs.LG

|last=Kernc

}}</ref> to obtain better results in fewer experiments than grid search and random search, due to the ability to reason about the quality of experiments before they are run.

|title=SAMBO: Sequential And Model-Based Optimization: Efficient global optimization in Python

|date=2024

|url=https://zenodo.org/records/14461363

|access-date=2025-01-30

|doi=10.5281/zenodo.14461363

}}</ref> to obtain better results in fewer evaluations compared to grid search and random search, due to the ability to reason about the quality of experiments before they are run.

=== Gradient-based optimization ===

For specific learning algorithms, it is possible to compute the gradient with respect to hyperparameters and then optimize the hyperparameters using [[gradient descent]]. The first usage of these techniques was focused on neural networks.<ref>{{cite journalbook |last1=Larsen|first1=Jan|last2= Hansen |first2=Lars Kai|last3=Svarer|first3=Claus|last4=Ohlsson|first4=M|title=Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop |chapter=Design and regularization of neural networks: theThe optimal use of a validation set |journaldate=Proceedings of the 1996 IEEE Signal Processing Society Workshop|datepages=62–71|doi=10.1109/NNSP.1996.548336|isbn=0-7803-3550-3|citeseerx=10.1.1.415.3266|s2cid=238874|chapter-url=http://orbit.dtu.dk/files/4545571/Svarer.pdf}}</ref> Since then, these methods have been extended to other models such as [[support vector machine]]s<ref>{{cite journal |author1=Olivier Chapelle |author2=Vladimir Vapnik |author3=Olivier Bousquet |author4=Sayan Mukherjee |title=Choosing multiple parameters for support vector machines |journal=Machine Learning |year=2002 |volume=46 |pages=131–159 |url=http://www.chapelle.cc/olivier/pub/mlj02.pdf | doi = 10.1023/a:1012450327387 |doi-access=free }}</ref> or logistic regression.<ref>{{cite journal |author1 =Chuong B|author2= Chuan-Sheng Foo|author3=Andrew Y Ng|journal = Advances in Neural Information Processing Systems |volume=20|title = Efficient multiple hyperparameter learning for log-linear models|year =2008|url=http://papers.nips.cc/paper/3286-efficient-multiple-hyperparameter-learning-for-log-linear-models.pdf}}</ref>

A different approach in order to obtain a gradient with respect to hyperparameters consists in differentiating the steps of an iterative optimization algorithm using [[automatic differentiation]].<ref>{{cite journal|last1=Domke|first1=Justin|title=Generic Methods for Optimization-Based Modeling|journal=Aistats|date=2012|volume=22|url=http://www.jmlr.org/proceedings/papers/v22/domke12/domke12.pdf|access-date=2017-12-09|archive-date=2014-01-24|archive-url=https://web.archive.org/web/20140124182520/http://jmlr.org/proceedings/papers/v22/domke12/domke12.pdf|url-status=dead}}</ref><ref name=abs1502.03492>{{cite arXiv |last1=Maclaurin|first1=DouglasDougal|last2=Duvenaud|first2=David|last3=Adams|first3=Ryan P.|eprint=1502.03492|title=Gradient-based Hyperparameter Optimization through Reversible Learning|class=stat.ML|date=2015}}</ref><ref>{{cite journal |last1=Franceschi |first1=Luca |last2=Donini |first2=Michele |last3=Frasconi |first3=Paolo |last4=Pontil |first4=Massimiliano |title=Forward and Reverse Gradient-Based Hyperparameter Optimization |journal=Proceedings of the 34th International Conference on Machine Learning |date=2017 |arxiv=1703.01785 |bibcode=2017arXiv170301785F |url=http://proceedings.mlr.press/v70/franceschi17a/franceschi17a-supp.pdf}}</ref><ref>{{cite arXiv | eprint=1810.10667 | last1=Shaban | first1=Amirreza | last2=Cheng | first2=Ching-An | last3=Hatch | first3=Nathan | last4=Boots | first4=Byron | title=Truncated Back-propagation for Bilevel Optimization | date=2018 | class=cs.LG }}</ref> A more recent work along this direction uses the [[implicit function theorem]] to calculate hypergradients and proposes a stable approximation of the inverse Hessian. The method scales to millions of hyperparameters and requires constant memory.<ref>{{cite arXiv | eprint=1911.02590 | last1=Lorraine | first1=Jonathan | last2=Vicol | first2=Paul | last3=Duvenaud | first3=David | title=Optimizing Millions of Hyperparameters by Implicit Differentiation | date=2019 | class=cs.LG }}</ref>

In a different approach,<ref>{{cite arXiv | eprint=1802.09419 | last1=Lorraine | first1=Jonathan | last2=Duvenaud | first2=David | title=Stochastic Hyperparameter Optimization through Hypernetworks | date=2018 | class=cs.LG }}</ref> a hypernetwork is trained to approximate the best response function. One of the advantages of this method is that it can handle discrete hyperparameters as well. Self-tuning networks<ref>{{cite arXiv | eprint=1903.03088 | last1=MacKay | first1=Matthew | last2=Vicol | first2=Paul | last3=Lorraine | first3=Jon | last4=Duvenaud | first4=David | last5=Grosse | first5=Roger | title=Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions | date=2019 | class=cs.LG }}</ref> offer a memory efficient version of this approach by choosing a compact representation for the hypernetwork. More recently, Δ-STN<ref>{{cite arXiv | eprint=2010.13514 | last1=Bae | first1=Juhan | last2=Grosse | first2=Roger | title=Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians | date=2020 | class=cs.LG }}</ref> has improved this method further by a slight reparameterization of the hypernetwork which speeds up training. Δ-STN also yields a better approximation of the best-response Jacobian by linearizing the network in the weights, hence removing unnecessary nonlinear effects of large changes in the weights.

Apart from hypernetwork approaches, gradient-based methods can be used to optimize discrete hyperparameters also by adopting a continuous relaxation of the parameters.<ref>{{cite arXiv | eprint=1806.09055 | last1=Liu | first1=Hanxiao | last2=Simonyan | first2=Karen | last3=Yang | first3=Yiming | title=DARTS: Differentiable Architecture Search | date=2018 | class=cs.LG }}</ref> Such methods have been extensively used for the optimization of architecture hyperparameters in [[neural architecture search]].

=== Evolutionary optimization ===

{{main article|Evolutionary algorithm}}

Evolutionary optimization is a methodology for the global optimization of noisy black-box functions. In hyperparameter optimization, evolutionary optimization uses [[evolutionary algorithms]] to search the space of hyperparameters for a given algorithm.<ref name="bergstra11" /> Evolutionary hyperparameter optimization follows a [[Evolutionary_algorithmEvolutionary algorithm#Implementation|process]] inspired by the biological concept of [[evolution]]:

# Create an initial population of random solutions (i.e., randomly generate tuples of hyperparameters, typically 100+)

# Evaluate the hyperparametershyperparameter tuples and acquire their [[fitness|fitness function]] (e.g., 10-fold [[Cross-validation (statistics)|cross-validation]] accuracy of the machine learning algorithm with those hyperparameters)

# Rank the hyperparameter tuples by their relative fitness

# Replace the worst-performing hyperparameter tuples with new hyperparameter tuplesones generated throughvia [[crossover (genetic algorithm)|crossover]] and [[mutation (genetic algorithm)|mutation]]

# Repeat steps 2-4 until satisfactory algorithm performance is reached or algorithm performance is no longer improving.

Evolutionary optimization has been used in hyperparameter optimization for statistical machine learning algorithms,<ref name="bergstra11" />, [[automated machine learning]], typical neural network <ref name="tpot1kousiouris1">{{cite journal |vauthors=Kousiouris G, Cuccinotta T, Varvarigou T | year = 2011 | title= The effects of scheduling, workload type and consolidation scenarios on virtual machine performance and their prediction through optimized artificial neural networks | url= https:/><ref/www.sciencedirect.com/science/article/abs/pii/S0164121211000951 | journal name="tpot2" Journal of Systems and Software | volume = 84 | issue = 8 | pages = 1270–1291| doi = 10.1016/j.jss.2011.04.013 | hdl = 11382/361472 | hdl-access = free }}</ref>, and [[Deep_learningDeep learning#Deep_neural_networksDeep neural networks|deep neural network]] architecture search,<ref name="miikkulainen1">{{cite arxivarXiv | vauthors = Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan A, Duffy N, Hodjat B | year = 2017 | title = Evolving Deep Neural Networks |eprint=1703.00548| class = cs.NE }}</ref><ref name="jaderberg1">{{cite arxivarXiv | vauthors = Jaderberg M, Dalibard V, Osindero S, Czarnecki WM, Donahue J, Razavi A, Vinyals O, Green T, Dunning I, Simonyan K, Fernando C, Kavukcuoglu K | year = 2017 | title = Population Based Training of Neural Networks |eprint=1711.09846| class = cs.LG }}</ref>, as well as training of the weights in deep neural networks.<ref name="such1">{{cite arxivarXiv | vauthors = Such FP, Madhavan V, Conti E, Lehman J, Stanley KO, Clune J | year = 2017 | title = Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning |eprint=1712.06567| class = cs.NE }}</ref>.

=== OthersPopulation-based ===

Population Based Training (PBT) learns both hyperparameter values and network weights. Multiple learning processes operate independently, using different hyperparameters. As with evolutionary methods, poorly performing models are iteratively replaced with models that adopt modified hyperparameter values and weights based on the better performers. This replacement model warm starting is the primary differentiator between PBT and other evolutionary methods. PBT thus allows the hyperparameters to evolve and eliminates the need for manual hypertuning. The process makes no assumptions regarding model architecture, loss functions or training procedures.

[[Radial basis function|RBF]]<ref name=abs1705.08520>{{cite arxiv |eprint=1705.08520|last1=Diaz|first1=Gonzalo|title=An effective algorithm for hyperparameter optimization of neural networks|last2=Fokoue|first2=Achille|last3=Nannicini|first3=Giacomo|last4=Samulowitz|first4=Horst|class=cs.AI|year=2017}}</ref> and [[spectral method|spectral]]<ref name=abs1706.00764>{{cite arxiv |eprint=1706.00764|last1=Hazan|first1=Elad|title=Hyperparameter Optimization: A Spectral Approach|last2=Klivans|first2=Adam|last3=Yuan|first3=Yang|class=cs.LG|year=2017}}</ref> approaches have also been developed.

PBT and its variants are adaptive methods: they update hyperparameters during the training of the models. On the contrary, non-adaptive methods have the sub-optimal strategy to assign a constant set of hyperparameters for the whole training.<ref>{{cite arXiv|last1=Li|first1=Ang|last2=Spyra|first2=Ola|last3=Perel|first3=Sagi|last4=Dalibard|first4=Valentin|last5=Jaderberg|first5=Max|last6=Gu|first6=Chenjie|last7=Budden|first7=David|last8=Harley|first8=Tim|last9=Gupta|first9=Pramod|date=2019-02-05|title=A Generalized Framework for Population Based Training|eprint=1902.01894|class=cs.AI}}</ref>

== Software ==

===Grid searchEarly stopping-based ===

[[File:Successive-halving-for-eight-arbitrary-hyperparameter-configurations.png|thumb|Successive halving for eight arbitrary hyperparameter configurations. The approach starts with eight models with different configurations and consecutively applies successive halving until only one model remains.]]

* [[LIBSVM]] comes with scripts for performing grid search.

A class of early stopping-based hyperparameter optimization algorithms is purpose built for large search spaces of continuous and discrete hyperparameters, particularly when the computational cost to evaluate the performance of a set of hyperparameters is high. Irace implements the iterated racing algorithm, that focuses the search around the most promising configurations, using statistical tests to discard the ones that perform poorly.<ref name="irace">{{cite journal |last1=López-Ibáñez |first1=Manuel |last2=Dubois-Lacoste |first2=Jérémie |last3=Pérez Cáceres |first3=Leslie |last4=Stützle |first4=Thomas |last5=Birattari |first5=Mauro |date=2016 |title=The irace package: Iterated Racing for Automatic Algorithm Configuration |journal=Operations Research Perspective |volume=3 |issue=3 |pages=43–58 |doi=10.1016/j.orp.2016.09.002|doi-access=free |hdl=10419/178265 |hdl-access=free }}</ref><ref name="race">{{cite journal |last1=Birattari |first1=Mauro |last2=Stützle |first2=Thomas |last3=Paquete |first3=Luis |last4=Varrentrapp |first4=Klaus |date=2002 |title=A Racing Algorithm for Configuring Metaheuristics |journal=Gecco 2002 |pages=11–18}}</ref>

* [[scikit-learn]] is a Python package which includes [http://scikit-learn.sourceforge.net/modules/grid_search.html grid] search.

Another early stopping hyperparameter optimization algorithm is successive halving (SHA),<ref>{{cite arXiv|last1=Jamieson|first1=Kevin|last2=Talwalkar|first2=Ameet|date=2015-02-27|title=Non-stochastic Best Arm Identification and Hyperparameter Optimization|eprint=1502.07943|class=cs.LG}}</ref> which begins as a random search but periodically prunes low-performing models, thereby focusing computational resources on more promising models. Asynchronous successive halving (ASHA)<ref>{{cite arXiv|last1=Li|first1=Liam|last2=Jamieson|first2=Kevin|last3=Rostamizadeh|first3=Afshin|last4=Gonina|first4=Ekaterina|last5=Hardt|first5=Moritz|last6=Recht|first6=Benjamin|last7=Talwalkar|first7=Ameet|date=2020-03-16|title=A System for Massively Parallel Hyperparameter Tuning|class=cs.LG|eprint=1810.05934v5}}</ref> further improves upon SHA's resource utilization profile by removing the need to synchronously evaluate and prune low-performing models. Hyperband<ref>{{cite journal|last1=Li|first1=Lisha|last2=Jamieson|first2=Kevin|last3=DeSalvo|first3=Giulia|last4=Rostamizadeh|first4=Afshin|last5=Talwalkar|first5=Ameet|date=2020-03-16|title=Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization|journal=Journal of Machine Learning Research|volume=18|pages=1–52|arxiv=1603.06560}}</ref> is a higher level early stopping-based algorithm that invokes SHA or ASHA multiple times with varying levels of pruning aggressiveness, in order to be more widely applicable and with fewer required inputs.

===Random searchOthers ===

[[Radial basis function|RBF]]<ref name=abs1705.08520>{{cite arXiv |eprint=1705.08520|last1=Diaz|first1=Gonzalo|title=An effective algorithm for hyperparameter optimization of neural networks|last2=Fokoue|first2=Achille|last3=Nannicini|first3=Giacomo|last4=Samulowitz|first4=Horst|class=cs.AI|year=2017}}</ref> and [[spectral method|spectral]]<ref name=abs1706.00764>{{cite arXiv |eprint=1706.00764|last1=Hazan|first1=Elad|title=Hyperparameter Optimization: A Spectral Approach|last2=Klivans|first2=Adam|last3=Yuan|first3=Yang|class=cs.LG|year=2017}}</ref> approaches have also been developed.

* [https://github.com/hyperopt/hyperopt hyperopt] and [https://github.com/hyperopt/hyperopt-sklearn hyperopt-sklearn] are Python packages which include random search.

* [[scikit-learn]] is a Python package which includes [http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.RandomizedSearchCV.html random] search.

* [http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html H2O AutoML] provides automated data preparation, hyperparameter tuning via random search, and stacked ensembles in a distributed machine learning platform.

== Issues with hyperparameter optimization ==

===Bayesian===

* [https://github.com/HIPS/Spearmint spearmint] Spearmint is a package to perform Bayesian optimization of machine learning algorithms.

* [https://rmcantin.bitbucket.io/html/ Bayesopt],<ref name="martinezcantin">{{cite journal

| title = BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits

| first1 = Ruben

| last1 = Martinez-Cantin

| journal = Journal of Machine Learning Research

| volume = 15

| year = 2014

| pages = 3915−3919

| url = http://jmlr.org/papers/volume15/martinezcantin14a/martinezcantin14a.pdf

| bibcode = 2014arXiv1405.7430M

| arxiv = 1405.7430

| class = cs.LG

}}</ref> an efficient implementation of Bayesian optimization in C/C++ with support for Python, Matlab and Octave.

* [https://github.com/yelp/MOE MOE] MOE is a Python/C++/CUDA library implementing Bayesian Global Optimization using Gaussian Processes.

* [http://www.cs.ubc.ca/labs/beta/Projects/autoweka/ Auto-WEKA]<ref name="autoweka">{{cite journal | vauthors = Kotthoff L, Thornton C, Hoos HH, Hutter F, Leyton-Brown K | year = 2017 | title = Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA | url = http://jmlr.org/papers/v18/16-261.html | journal = Journal of Machine Learning Research | pages = 1–5 }}</ref> is a Bayesian hyperparameter optimization layer on top of [[Weka (machine learning)|WEKA]].

* [https://github.com/automl/auto-sklearn Auto-sklearn]<ref name="autosklearn">{{cite journal | vauthors = Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F | year = 2015 | title = Efficient and Robust Automated Machine Learning | url = https://papers.nips.cc/paper/5872-efficient-and-robust-automated-machine-learning | journal = Advances in Neural Information Processing Systems 28 (NIPS 2015) | pages = 2962–2970 }}</ref> is a Bayesian hyperparameter optimization layer on top of [[scikit-learn]].

When hyperparameter optimization is done, the set of hyperparameters are often fitted on a training set and selected based on the generalization performance, or score, of a validation set. However, this procedure is at risk of overfitting the hyperparameters to the validation set. Therefore, the generalization performance score of the validation set (which can be several sets in the case of a cross-validation procedure) cannot be used to simultaneously estimate the generalization performance of the final model. In order to do so, the generalization performance has to be evaluated on a set independent (which has no intersection) of the set (or sets) used for the optimization of the hyperparameters, otherwise the performance might give a value which is too optimistic (too large). This can be done on a second test set, or through an outer [[Cross-validation (statistics)|cross-validation]] procedure called nested cross-validation, which allows an unbiased estimation of the generalization performance of the model, taking into account the bias due to the hyperparameter optimization.

===Gradient based===

* [https://github.com/HIPS/hypergrad hypergrad] is a Python package for differentiation with respect to hyperparameters.<ref name=abs1502.03492/>

===Evolutionary===

* [https://github.com/rhiever/tpot TPOT]<ref name="tpot1">{{cite journal | vauthors = Olson RS, Urbanowicz RJ, Andrews PC, Lavender NA, Kidd L, Moore JH | year = 2016 | title = Automating biomedical data science through tree-based pipeline optimization | url = https://link.springer.com/chapter/10.1007/978-3-319-31204-0_9 | journal = Proceedings of EvoStar 2016 | volume = 9597 | pages = 123–137 | doi = 10.1007/978-3-319-31204-0_9 | series = Lecture Notes in Computer Science | isbn = 978-3-319-31203-3 }}</ref><ref name="tpot2">{{cite journal | vauthors = Olson RS, Bartley N, Urbanowicz RJ, Moore JH | year = 2016 | title = Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science | url = https://dl.acm.org/citation.cfm?id=2908918 | journal = Proceedings of EvoBIO 2016 | pages = 485–492 | doi = 10.1145/2908812.2908918 | isbn = 9781450342063 }}</ref> is a Python package that automatically creates and optimizes full machine learning pipelines using [[genetic programming]].

* [https://github.com/joeddav/devol devol] is a Python package that performs Deep Neural Network architecture search using [[genetic programming]].

===Other===

* [https://github.com/hyperopt/hyperopt hyperopt] and [https://github.com/hyperopt/hyperopt-sklearn hyperopt-sklearn] are Python packages which include [[kernel density estimation|Tree of Parzen Estimators]] based distributed hyperparameter optimization.

* [https://github.com/CMA-ES/pycma pycma] is a Python implementation of [[CMA-ES|Covariance Matrix Adaptation Evolution Strategy]].

* [http://sumo.intec.ugent.be SUMO-Toolbox]<ref name="gorissen">{{cite journal

| title = A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design

| first1 = Dirk

| last1 = Gorissen

| first2 = Karel

| last2 = Crombecq

| first3 = Ivo

| last3 = Couckuyt

| first4 = Piet

| last4 = Demeester

| first5 = Tom

| last5 = Dhaene

| journal = J. Machine Learning Research

| volume = 11

| year = 2010

| pages = 2051–2055

| url = http://www.jmlr.org/papers/volume11/gorissen10a/gorissen10a.pdf

}}</ref> is a [[MATLAB]] toolbox for [[surrogate model]]ing supporting a wide collection of hyperparameter optimization algorithm for many model types.

* [https://github.com/coin-or/rbfopt rbfopt] is a Python package that uses a [[radial basis function]] model<ref name=abs1705.08520/>

* [https://github.com/callowbird/Harmonica Harmonica] is a Python package for spectral hyperparameter optimization.<ref name=abs1706.00764/>

===Multiple===

* [https://github.com/mlr-org/mlr mlr] is a [[R]] package that contains a large number of different hyperparameter optimization techniques.

== See also ==

* [[Automated machine learning]] (AutoML)

* [[Neural architecture search]]

* [[Bias-variance dilemma]]

* [[Dimensionality reduction]]

* [[Feature selection]]

* [[Meta-optimization]]

* [[Model selection]]

* [[Self-tuning]]

* [[XGBoost]]

* [[Optuna]]

== References ==

{{Reflist|30em}}

{{Differentiable computing}}

[[Category:Machine learning]]