Content deleted Content added
SimLibrarian (talk | contribs) |
m Open access bot: url-access=subscription updated in citation with #oabot. |
||
| (22 intermediate revisions by 14 users not shown) | |||
Line 1:
{{Short description|Machine learning that combines deep learning and reinforcement learning}}
{{Machine learning}}
'''Deep reinforcement learning''' ('''deep RL''') is a subfield of [[machine learning]] that combines [[reinforcement learning]] (RL) and [[deep learning]]. RL considers the problem of a computational agent learning to make decisions by trial and error. Deep RL incorporates deep learning into the solution, allowing agents to make decisions from unstructured input data without manual engineering of the [[state space]]. Deep RL algorithms are able to take in very large inputs (e.g. every pixel rendered to the screen in a video game) and decide what actions to perform to optimize an objective (e.g. maximizing the game score). Deep reinforcement learning has been used for a diverse set of applications including but not limited to [[robotics]], [[video game]]s, [[natural language processing]], [[computer vision]],<ref>{{Cite journal |
== Overview ==
Line 7:
=== Deep learning ===
[[File:Neural_network_example.svg|thumb|241x241px|Depiction of a basic artificial neural network]]
[[Deep learning]] is a form of [[machine learning]] that utilizes a neural network to transform a set of inputs into a set of outputs via an [[artificial neural network]]. Deep learning methods, often using [[supervised learning]] with labeled datasets, have been shown to solve tasks that involve handling complex, high-dimensional raw input data (such as images) with less manual [[feature engineering]] than prior methods, enabling significant progress in several fields including [[computer vision]] and [[natural language processing]]. In the past decade, deep RL has achieved remarkable results on a range of problems, from single and multiplayer games such as [[Go (game)|Go]], [[Atari Games]], and ''[[Dota 2
=== Reinforcement learning ===
Line 29:
Deep reinforcement learning reached another milestone in 2015 when [[AlphaGo]],<ref name="AlphaGo"/> a computer program trained with deep RL to play [[Go (game)|Go]], became the first computer Go program to beat a human professional Go player without handicap on a full-sized 19×19 board.
In a subsequent project in 2017, [[AlphaZero]] improved performance on Go while also demonstrating they could use the same algorithm to learn to play [[chess]] and [[shogi]] at a level competitive or superior to existing computer programs for those games, and again improved in 2019 with [[MuZero]].<ref name="muzero"/> Separately, another milestone was achieved by researchers from [[Carnegie Mellon University]] in 2019 developing [[Pluribus (poker bot)|Pluribus]], a computer program to play [[poker]] that was the first to beat professionals at multiplayer games of no-limit [[Texas hold 'em]]. [[OpenAI Five]], a program for playing five-on-five ''[[Dota 2
Deep reinforcement learning has also been applied to many domains beyond games. In robotics, it has been used to let robots perform simple household tasks<ref name="levine2016"/> and solve a Rubik's cube with a robot hand.<ref name="openaihand"/><ref name="openaihandarxiv"/> Deep RL has also found sustainability applications, used to reduce energy consumption at data centers.<ref name="deepmindcooling"/> Deep RL for [[autonomous driving]] is an active area of research in academia and industry.<ref name="neurips2021ml4ad"/> [[Loon_LLC|Loon]] explored deep RL for autonomously navigating their high-altitude balloons.<ref name="loonrl"/>
Line 45:
=== Exploration ===
An RL agent must balance the exploration/exploitation tradeoff: the problem of deciding whether to pursue actions that are already known to yield high rewards or explore other actions in order to discover higher rewards. RL agents usually collect data with some type of stochastic policy, such as a [[Boltzmann distribution]] in discrete action spaces or a [[Normal distribution|Gaussian distribution]] in continuous action spaces, inducing basic exploration behavior. The idea behind novelty-based, or curiosity-driven, exploration is giving the agent a motive to explore unknown outcomes in order to find the best solutions. This is done by "modify[ing] the loss function (or even the network architecture) by adding terms to incentivize exploration".<ref>{{cite book|last1=Reizinger|first1=Patrik|last2=Szemenyei|first2=Márton|date=2019-10-23|title=ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)|chapter=Attention-Based Curiosity-Driven Exploration in Deep Reinforcement Learning |pages=3542–3546 |doi=10.1109/ICASSP40776.2020.9054546 |arxiv=1910.10840|isbn=978-1-5090-6631-5 |s2cid=204852215 }}</ref> An agent may also be aided in exploration by utilizing demonstrations of successful trajectories, or reward-shaping, giving an agent intermediate rewards that are customized to fit the task it is attempting to complete.<ref>{{Citation|last=Wiewiora|first=Eric|title=Reward Shaping|date=2010|url=https://doi.org/10.1007/978-0-387-30164-8_731|encyclopedia=Encyclopedia of Machine Learning|pages=863–865|editor-last=Sammut|editor-first=Claude|place=Boston, MA|publisher=Springer US|language=en|doi=10.1007/978-0-387-30164-8_731|isbn=978-0-387-30164-8|access-date=2020-11-16|editor2-last=Webb|editor2-first=Geoffrey I.|url-access=subscription}}</ref>
=== Off-policy reinforcement learning ===
Line 73:
<ref name="francoislavet2018">{{Cite journal|last1=Francois-Lavet|first1=Vincent|last2=Henderson|first2=Peter|last3=Islam|first3=Riashat|last4=Bellemare|first4=Marc G.|last5=Pineau|first5=Joelle|date=2018|title=An Introduction to Deep Reinforcement Learning|journal=Foundations and Trends in Machine Learning|volume=11|issue=3–4|pages=219–354|arxiv=1811.12560|bibcode=2018arXiv181112560F|doi=10.1561/2200000071|issn=1935-8237|s2cid=54434537}}</ref>
<ref name="Hassabis">{{cite speech |last1=Demis |first1=Hassabis | date=March 11, 2016 |title= Artificial Intelligence and the Future. |url= https://www.youtube.com/watch?v=8Z2eLTSCuBk}}</ref>
<ref name="TD-Gammon">{{cite journal
<ref name="sutton1996">{{cite book |last1=Sutton |first1=Richard |last2=Barto |first2=Andrew |date=September 1996 |title=Reinforcement Learning: An Introduction |publisher=Athena Scientific}}</ref>
<ref name="tsitsiklis1996">{{cite book |last1=Bertsekas |first2=Dimitri |last2=Tsitsiklis |first1=John |date=September 1996 |title=Neuro-Dynamic Programming |url=http://athenasc.com/ndpbook.html |publisher=Athena Scientific |isbn=1-886529-10-8}}</ref>
Line 86:
<ref name="AlphaGo">{{Cite journal|title = Mastering the game of Go with deep neural networks and tree search|journal = [[Nature (journal)|Nature]]| issn= 0028-0836|pages = 484–489|volume = 529|issue = 7587|doi = 10.1038/nature16961|pmid = 26819042|first1 = David|last1 = Silver|author-link1=David Silver (programmer)|first2 = Aja|last2 = Huang|author-link2=Aja Huang|first3 = Chris J.|last3 = Maddison|first4 = Arthur|last4 = Guez|first5 = Laurent|last5 = Sifre|first6 = George van den|last6 = Driessche|first7 = Julian|last7 = Schrittwieser|first8 = Ioannis|last8 = Antonoglou|first9 = Veda|last9 = Panneershelvam|first10= Marc|last10= Lanctot|first11= Sander|last11= Dieleman|first12=Dominik|last12= Grewe|first13= John|last13= Nham|first14= Nal|last14= Kalchbrenner|first15= Ilya|last15= Sutskever|author-link15=Ilya Sutskever|first16= Timothy|last16= Lillicrap|first17= Madeleine|last17= Leach|first18= Koray|last18= Kavukcuoglu|first19= Thore|last19= Graepel|first20= Demis |last20=Hassabis|author-link20=Demis Hassabis|date= 28 January 2016|bibcode = 2016Natur.529..484S|s2cid = 515925}}{{closed access}}</ref>
<ref name="levine2016">{{Cite journal |last1=Levine |first1=Sergey |last2=Finn |first2=Chelsea |author-link2=Chelsea Finn |last3=Darrell |first3=Trevor |last4=Abbeel |first4=Pieter |date=January 2016 |title=End-to-end training of deep visuomotor policies |url=https://www.jmlr.org/papers/volume17/15-389/15-389.pdf |journal=JMLR |volume=17 |arxiv=1504.00702}}</ref>
<ref name="openaihand">{{Cite web|title=OpenAI - Solving Rubik's Cube With A Robot Hand|url=https://openai.com/blog/solving-rubiks-cube/|website=OpenAI|date=5 January 2021 }}</ref>
<ref name="openaihandarxiv">{{Cite conference|title= Solving Rubik's Cube with a Robot Hand |last1=OpenAI |display-authors=etal|date=2019|arxiv=1910.07113 }}</ref>
<ref name="deepmindcooling">{{Cite web|title=DeepMind AI Reduces Google Data Centre Cooling Bill by 40% |url=https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40|website=DeepMind|date=14 May 2024 }}</ref>
<ref name="neurips2021ml4ad">{{Cite web|title=Machine Learning for Autonomous Driving Workshop @ NeurIPS 2021|url=https://ml4ad.github.io/|website=NeurIPS 2021|date=December 2021}}</ref>
<ref name="williams1992">{{Cite journal|last1=Williams|first1=Ronald J|journal=Machine Learning|pages=229–256|title = Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning|date=1992|volume=8|issue=3–4|doi=10.1007/BF00992696|s2cid=2332513|doi-access=free}}</ref>
Line 99:
<ref name="schaul2015uva">{{Cite conference| title=Universal Value Function Approximators|last1=Schaul|first1=Tom |last2=Horgan|first2=Daniel |last3=Gregor|first3=Karol |last4=Silver|first4=David |conference=International Conference on Machine Learning (ICML) |date=2015| url=http://proceedings.mlr.press/v37/schaul15.html}}</ref>
<ref name="muzero">{{cite journal |last1=Schrittwieser |first1=Julian |last2=Antonoglou |first2=Ioannis |last3=Hubert |first3=Thomas |last4=Simonyan |first4=Karen |last5=Sifre |first5=Laurent |last6=Schmitt |first6=Simon |last7=Guez |first7=Arthur |last8=Lockhart |first8=Edward |last9=Hassabis |first9=Demis |last10=Graepel |first10=Thore |last11=Lillicrap |first11=Timothy |last12=Silver |first12=David |title=Mastering Atari, Go, chess and shogi by planning with a learned model |journal=Nature |date=23 December 2020 |volume=588 |issue=7839 |pages=604–609 |doi=10.1038/s41586-020-03051-4 |pmid=33361790 |url=https://www.nature.com/articles/s41586-020-03051-4|arxiv=1911.08265 |bibcode=2020Natur.588..604S |s2cid=208158225 }}</ref>
<ref name="loonrl">{{cite journal |last1=Bellemare |first1=Marc |last2=Candido |first2=Salvatore |last3=Castro |first3=Pablo |last4=Gong |first4=Jun |last5=Machado |first5=Marlos |last6=Moitra |first6=Subhodeep |last7=Ponda |first7=Sameera |last8=Wang |first8=Ziyu |title=Autonomous navigation of stratospheric balloons using reinforcement learning |journal=Nature |date=2 December 2020 |volume=588 |issue=7836 |pages=77–82 |doi=10.1038/s41586-020-2939-8 |pmid=33268863 |bibcode=2020Natur.588...77B |s2cid=227260253 |url=https://www.nature.com/articles/s41586-020-2939-8|url-access=subscription }}</ref>
<ref name="deepirl">{{cite arXiv| last1=Wulfmeier|first1=Markus|last2=Ondruska|first2=Peter|last3=Posner|first3=Ingmar|date=2015|title= Maximum Entropy Deep Inverse Reinforcement Learning |class=cs.LG|eprint=1507.04888}}</ref>
</references>
| |||