Bayesian optimization: Difference between revisions

Revision as of 09:25, 4 August 2025

Revision as of 12:11, 21 August 2025

Line 22:

==== From theory to practice ====

In the 1990s, Bayesian optimization began to gradually transition from pure theory to real-world applications. In 1998, Donald R. Jones<ref>{{Cite web |title=Donald R. Jones |url=https://scholar.google.com/citations?user=CZhZ4MYAAAAJ&hl=en |access-date=2025-02-25 |website=scholar.google.com}}</ref> and his coworkers published a paper titled “Gaussian“Efficient Global Optimization of Expensive Black-Box Functions<ref>{{Cite book |last=GrcarJones |first=JosephDonald FR. |url=https://link.springer.com/article/10.1023/A:1008306431147 |title=MathematiciansEfficient Global Optimization of GaussianExpensive EliminationBlack-Box Functions |last2=Schonlau |first2=Matthias |last3=Welch |first3=William J. |year=1998}}</ref>”. In this paper, they proposed the Gaussian Process (GP) and elaborated on the Expected Improvement principle (EI) proposed by Jonas Mockus in 1978. Through the efforts of Donald R. Jones and his colleagues, Bayesian Optimization began to shine in the fields like computers science and engineering. However, the computational complexity of Bayesian optimization for the computing power at that time still affected its development to a large extent.

In the 21st century, with the gradual rise of artificial intelligence and bionic robots, Bayesian optimization has been widely used in machine learning and deep learning, and has become an important tool for [[Hyperparameter optimization|Hyperparameter Tuning]].<ref>T. T. Joy, S. Rana, S. Gupta and S. Venkatesh, "Hyperparameter tuning for big data using Bayesian optimisation," 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico, 2016, pp. 2574-2579, doi: 10.1109/ICPR.2016.7900023. keywords: {Big Data;Bayes methods;Optimization;Tuning;Data models;Gaussian processes;Noise measurement},</ref> Companies such as Google, Facebook and OpenAI have added Bayesian optimization to their deep learning frameworks to improve search efficiency. However, Bayesian optimization still faces many challenges, for example, because of the use of Gaussian Process<ref>{{Cite book |title=Neural Networks and Machine Learning|contribution=Introduction to Gaussian processes|first=D. J. C.|last=Mackay|editor-first=C. M.|editor-last=Bishop |contribution-url=https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=e045b76dc5daf9f4656ac10b456c5d1d9de5bc84 |archive-url=http://web.archive.org/web/20240423144014/https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=e045b76dc5daf9f4656ac10b456c5d1d9de5bc84 |archive-date=2024-04-23 |access-date=2025-03-06 |series=NATO ASI Series|volume=168|pages=133–165|year=1998}}</ref> as a proxy model for optimization, when there is a lot of data, the training of Gaussian Process will be very slow and the computational cost is very high. This makes it difficult for this optimization method to work well in more complex drug development and medical experiments.