|
=== Others ===
[[Radial basis function|RBF]]<ref name=abs1705.08520>{{cite arXiv |eprint=1705.08520|last1=Diaz|first1=Gonzalo|title=An effective algorithm for hyperparameter optimization of neural networks|last2=Fokoue|first2=Achille|last3=Nannicini|first3=Giacomo|last4=Samulowitz|first4=Horst|class=cs.AI|year=2017}}</ref> and [[spectral method|spectral]]<ref name=abs1706.00764>{{cite arXiv |eprint=1706.00764|last1=Hazan|first1=Elad|title=Hyperparameter Optimization: A Spectral Approach|last2=Klivans|first2=Adam|last3=Yuan|first3=Yang|class=cs.LG|year=2017}}</ref> approaches have also been developed.
== Open-source software ==
===Grid search===
*[https://github.com/determined-ai/determined Determined], a DL Training Platform includes grid search for PyTorch and TensorFlow (Keras and Estimator) models.
*[http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html H2O AutoML] provides grid search over algorithms in the H2O open source machine learning library.
*[https://github.com/kubeflow/katib Katib] is a Kubernetes-native system that includes grid search.
*[[scikit-learn]] is a Python package that includes [http://scikit-learn.sourceforge.net/modules/grid_search.html grid] search.
*[https://github.com/autonomio/talos Talos] includes grid search for [[Keras]].
*[https://ray.readthedocs.io/en/latest/tune.html Tune] is a Python library for distributed hyperparameter tuning and supports grid search.
*[https://orion.readthedocs.io/ Oríon] is an asynchronous framework for distributed black-box optimization and hyperparameter optimization that includes grid search.
*[https://github.com/awslabs/syne-tune Syne Tune] is a Python library for asynchronous hyperparameter and architecture optimization that supports grid search.
===Random search===
*[https://github.com/determined-ai/determined Determined] is a DL Training Platform that supports random search for PyTorch and TensorFlow (Keras and Estimator) models.
* [https://github.com/hyperopt/hyperopt hyperopt], also via [https://github.com/maxpumperla/hyperas hyperas] and [https://github.com/hyperopt/hyperopt-sklearn hyperopt-sklearn], are Python packages which include random search.
*[https://github.com/kubeflow/katib Katib] is a Kubernetes-native system that includes random search.
* [[scikit-learn]] is a Python package which includes [https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html random] search.
* [[caret]] is a R package which includes [http://topepo.github.io/caret/random-hyperparameter-search.html grid & random] search.
*[https://github.com/autonomio/talos Talos] includes a customizable random search for [[Keras]].
*[https://ray.readthedocs.io/en/latest/tune.html Tune] is a Python library for distributed hyperparameter tuning and supports random search over arbitrary parameter distributions.
*[https://orion.readthedocs.io/ Oríon] is an asynchronous framework for distributed black-box optimization and hyperparameter optimization that includes random search.
*[https://github.com/awslabs/syne-tune Syne Tune] is a Python library for asynchronous hyperparameter and architecture optimization that supports random search.
===Bayesian===
* [https://github.com/automl/auto-sklearn Auto-sklearn]<ref name="autosklearn">{{cite journal | vauthors = Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F | year = 2015 | title = Efficient and Robust Automated Machine Learning | url = https://papers.nips.cc/paper/5872-efficient-and-robust-automated-machine-learning | journal = Advances in Neural Information Processing Systems 28 (NIPS 2015) | volume = 28 | pages = 2962–2970 }}</ref> is a Bayesian hyperparameter optimization layer on top of [[scikit-learn]].
* [https://github.com/facebook/Ax Ax]<ref name=AxBoTorch>{{cite web |url=https://ai.facebook.com/blog/open-sourcing-ax-and-botorch-new-ai-tools-for-adaptive-experimentation/ |title=Open-sourcing Ax and BoTorch: New AI tools for adaptive experimentation |year=2019}}</ref> is a Python-based experimentation platform that supports Bayesian optimization and bandit optimization as exploration strategies.
* [https://github.com/baptistar/BOCS BOCS] is a Matlab package which uses [[semidefinite programming]] for minimizing a black-box function over discrete inputs.<ref name="arXiv:1806.08838">{{cite arXiv |year=2018 |title=Bayesian Optimization of Combinatorial Structures |eprint=1806.08838|last1=Baptista |first1=Ricardo |last2=Poloczek |first2=Matthias |class=stat.ML }}</ref> A Python 3 implementation is also included.
* [https://github.com/automl/HpBandSter HpBandSter] is a Python package which combines Bayesian optimization with bandit-based methods.<ref name="arXiv:1807.01774">{{cite arXiv |year=2018 |title=BOHB: Robust and Efficient Hyperparameter Optimization at Scale |eprint=1807.01774|last1=Falkner |first1=Stefan |last2=Klein |first2=Aaron |last3=Hutter |first3=Frank |class=stat.ML }}</ref>
*[https://github.com/kubeflow/katib Katib] is a Kubernetes-native system which includes bayesian optimization.
*[https://github.com/mlr-org/mlrMBO mlrMBO], also with [https://github.com/mlr-org/mlr mlr], is an [[R (programming language)|R]] package for model-based/Bayesian optimization of black-box functions.
*[https://optuna.readthedocs.io/en/latest/ optuna] is a Python package for black box optimization, compatible with arbitrary functions that need to be optimized.
* [https://github.com/scikit-optimize/scikit-optimize scikit-optimize] is a Python package or sequential model-based optimization with a scipy.optimize interface.<ref name=skopt>{{Cite web|url=https://scikit-optimize.github.io/|title=skopt API documentation|website=scikit-optimize.github.io}}</ref>
* [https://github.com/automl/SMAC3 SMAC] SMAC is a Python/Java library implementing Bayesian optimization.<ref name="SMAC">{{cite journal | vauthors = Hutter F, Hoos HH, Leyton-Brown K | title = Sequential Model-Based Optimization for General Algorithm Configuration | url = https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf | journal = Proceedings of the Conference on Learning and Intelligent OptimizatioN (LION 5)}}</ref>
* [https://github.com/PhilippPro/tuneRanger tuneRanger] is an R package for tuning random forests using model-based optimization.
* [https://github.com/google/vizier Open Source Vizier], a Python service which allows Bayesian Optimization development built upon [https://www.tensorflow.org/probability Tensorflow Probability].
*[https://orion.readthedocs.io/ Oríon] is a Python package that provides a built-in implementation of the Bayesian optimization algorithm TPE as well as integrations with [https://github.com/facebook/Ax Ax], [https://github.com/scikit-optimize/scikit-optimize scikit-optimize] and [https://github.com/automl/HpBandSter HpBandSter] for additional Bayesian optimization algorithms.
*[https://github.com/awslabs/syne-tune Syne Tune] is a Python library for asynchronous hyperparameter and architecture optimization that supports Bayesian optimization, as well as model-based multi-fidelity algorithms.
===Gradient-based optimization===
* [https://github.com/lucfra/FAR-HO FAR-HO] is a Python package containing Tensorflow implementations and wrappers for gradient-based hyperparameter optimization with forward and reverse mode algorithmic differentiation.
* [https://github.com/dmlc/xgboost XGBoost] is an open-source software library that provides a gradient boosting framework for C++, Java, Python, R, and Julia.
===Evolutionary===
* [https://github.com/DEAP/deap deap] is a Python framework for general evolutionary computation which is flexible and integrates with parallelization packages like [https://github.com/soravux/scoop scoop] and [[PySpark|pyspark]], and other Python frameworks like [[Scikit-learn|sklearn]] via [https://github.com/rsteca/sklearn-deap sklearn-deap].
*[https://github.com/determined-ai/determined Determined] is a DL Training Platform that supports PBT for optimizing PyTorch and TensorFlow (Keras and Estimator) models.
* [https://github.com/joeddav/devol devol] is a Python package that performs Deep Neural Network architecture search using [[genetic programming]].
* [https://github.com/facebookresearch/nevergrad nevergrad]<ref name=nevergrad_issue1/> is a Python package which includes [[Differential evolution]], [[Evolution strategy]], [[Bayesian optimization]], population control methods for the noisy case and [[Particle swarm optimization]].<ref name=nevergrad/>
*[https://ray.readthedocs.io/en/latest/tune.html Tune] is a Python library for distributed hyperparameter tuning and leverages [https://github.com/facebookresearch/nevergrad nevergrad] for evolutionary algorithm support.
* [https://github.com/google/vizier Open Source Vizier] is a Python service that supports algorithms such as NSGA-II<ref name="doi10.1109/4235.996017">{{Cite journal | doi = 10.1109/4235.996017| title = A fast and elitist multiobjective genetic algorithm: NSGA-II| journal = IEEE Transactions on Evolutionary Computation| volume = 6| issue = 2| pages = 182| year = 2002| last1 = Deb | first1 = K.| last2 = Pratap | first2 = A.| last3 = Agarwal | first3 = S.| last4 = Meyarivan | first4 = T.| citeseerx = 10.1.1.17.7771}}</ref> and integrated with [https://github.com/google/pyglove PyGlove] to allow evolutionary computing.
*[https://orion.readthedocs.io/ Oríon] is a Python package that includes the algorithm Population Based Training and Population Based Bandits, as well as an integration with [https://github.com/facebookresearch/nevergrad nevergrad] for additional evolutionary algorithms.
*[https://github.com/awslabs/syne-tune Syne Tune] is a Python library for asynchronous hyperparameter and architecture optimization that supports evolutionary algorithms.
===Early Stopping===
*[https://github.com/determined-ai/determined Determined] is a DL Training Platform that supports Hyperband for PyTorch and TensorFlow (Keras and Estimator) models.
* [https://iridia.ulb.ac.be/irace/ irace] is an R package that implements the iterated racing algorithm.<ref name="irace"/><ref name="race"/>
*[https://github.com/kubeflow/katib Katib] is a Kubernetes-native system that includes hyperband.
*[https://github.com/awslabs/syne-tune Syne Tune] is a Python library for asynchronous hyperparameter and architecture optimization that supports multi-fidelity algorithms (also known as early-stopping algorithms).
===Other===
*[https://github.com/determined-ai/determined Determined] is a DL Training Platform that supports random, grid, PBT, Hyperband and NAS approaches to hyperparameter optimization for PyTorch and TensorFlow (Keras and Estimator) models.
* [[dlib]]<ref name=dlib_github>{{Cite web|url=https://github.com/davisking/dlib|title=A toolkit for making real world machine learning and data analysis applications in C++: davisking/dlib|date=February 25, 2019|via=GitHub}}</ref> is a C++ package with a Python API which has a parameter-free optimizer based on [https://arxiv.org/abs/1703.02628 LIPO] and [[trust region]] optimizers working in tandem.<ref name=dlib_blog>{{cite web |last1=King |first1=Davis |title=A Global Optimization Algorithm Worth Using |url=http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html}}</ref>
* [https://github.com/callowbird/Harmonica Harmonica] is a Python package for spectral hyperparameter optimization.<ref name=abs1706.00764/>
* [https://github.com/hyperopt/hyperopt hyperopt], also via [https://github.com/maxpumperla/hyperas hyperas] and [https://github.com/hyperopt/hyperopt-sklearn hyperopt-sklearn], are Python packages which include [[tree of Parzen estimators]] based distributed hyperparameter optimization.
*[https://github.com/kubeflow/katib/ Katib] is a Kubernetes-native system which includes grid, random search, bayesian optimization, hyperband, and NAS based on reinforcement learning.
* [https://github.com/facebookresearch/nevergrad nevergrad]<ref name=nevergrad_issue1>{{Cite web|url=https://github.com/facebookresearch/nevergrad/issues/1|title=[QUESTION] How to use to optimize NN hyperparameters · Issue #1 · facebookresearch/nevergrad|website=GitHub}}</ref> is a Python package for gradient-free optimization using techniques such as differential evolution, sequential quadratic programming, fastGA, covariance matrix adaptation, population control methods, and particle swarm optimization.<ref name=nevergrad>{{Cite web|url=https://code.fb.com/ai-research/nevergrad/|title=Nevergrad: An open source tool for derivative-free optimization|date=December 20, 2018}}</ref>
* [[Neural Network Intelligence]] (NNI) is a Python package which includes hyperparameter tuning for neural networks in local and distributed environments. Its techniques include TPE, random, anneal, evolution, SMAC, batch, grid, and hyperband.
* [https://github.com/sherpa-ai/sherpa parameter-sherpa] is a similar Python package which includes several techniques grid search, Bayesian and genetic Optimization
* [https://github.com/wwu-mmll/photonai photonai] is a high level Python API for designing and optimizing machine learning pipelines based on grid, random search and bayesian optimization.
* [https://github.com/CMA-ES/pycma pycma] is a Python implementation of [[CMA-ES|Covariance Matrix Adaptation Evolution Strategy]].
* [https://github.com/coin-or/rbfopt rbfopt] is a Python package that uses a [[radial basis function]] model<ref name=abs1705.08520/>
* [https://ray.readthedocs.io/en/latest/tune.html Tune] is a Python library for hyperparameter tuning execution and integrates with/scales many existing hyperparameter optimization libraries such as [https://github.com/hyperopt/hyperopt hyperopt], [https://github.com/facebookresearch/nevergrad nevergrad], and [https://github.com/scikit-optimize/scikit-optimize scikit-optimize].
* [https://github.com/google/vizier Open Source Vizier] is a Python-based service for blackbox hyperparameter optimization and research, based on Google's internal Vizier service.<ref name=google_vizier>{{cite web |last1=Golovin |first1=Daniel |last2=Solnik |first2=Benjamin |last3=Subhodeep |first3=Moitra |last4=Kochanski |first4=Greg |last5=Karro |first5=John |last6=Sculley |first6=D. |title=Google Vizier: A Service for Black-Box Optimization |url=https://dl.acm.org/doi/10.1145/3097983.3098043}}</ref>
* [https://orion.readthedocs.io/ Oríon] is an asynchronous framework for distributed black-box optimization and hyperparameter optimization that includes built-in algorithms such as grid search, random search, MOFA, Hyperband, PBT, TPE and several integrations with hyperparameter optimization libraries such as [https://github.com/facebook/Ax Ax], [https://github.com/scikit-optimize/scikit-optimize scikit-optimize], [https://github.com/automl/HpBandSter HpBandSter], [https://github.com/huawei-noah/HEBO HEBO], [https://github.com/facebookresearch/nevergrad nevergrad] and [https://github.com/automl/DEHB DEHB].
== Commercial services ==
* [https://aws.amazon.com/sagemaker/automatic-model-tuning/ Amazon Sagemaker Automatic Model Tuning] offers grid search, random search, Gaussian process-based Bayesian optimization and asynchronous successive halving.
* [https://bigml.com/api/optimls BigML OptiML] supports mixed search domains
* [https://cloud.google.com/ml-engine/docs/tensorflow/using-hyperparameter-tuning Google Cloud Vertex Vizier] supports mixed search domains, multiobjective, multifidelity, and safety constraints.
* [https://indiesolver.com Indie Solver] supports multiobjective, multifidelity and constraint optimization
* [https://mindfoundry.ai/OPTaaS Mind Foundry OPTaaS] supports mixed search domains, multiobjective, constraints, parallel optimization and surrogate models.
* [https://sigopt.com SigOpt] supports mixed search domains, multiobjective, multisolution, multifidelity, constraint (linear and black-box), and parallel optimization.
== See also ==
| 