User:ZachsGenericUsername/sandbox/Deep reinforcement learning: Difference between revisions
Content deleted Content added
restructructured headings |
Citation bot (talk | contribs) Removed URL that duplicated identifier. | Use this bot. Report bugs. | Suggested by Eastmain | #UCB_webform 653/859 |
||
| (13 intermediate revisions by 2 users not shown) | |||
Line 3:
'''Deep reinforcement learning (DRL)''' is a [[machine learning]] method that takes principles from both [[reinforcement learning]] and [[deep learning]] to obtain benefits from both.
Deep reinforcement learning has a large diversity of applications including but not limited to video games, computer science, healthcare, and finance.
== Overview ==
=== Reinforcement Learning ===
[[File:Markov diagram v2.svg|alt=Diagram explaining the loop recurring in reinforcement learning algorithms|thumb|Diagram of the loop recurring in reinforcement learning algorithms]][[Reinforcement learning]] is a process in which an agent learns to preform an action through trial and error.
=== Deep Learning ===
[[File:Neural network example.svg|thumb|241x241px|Depiction of a basic artificial neural network]]
[[Deep learning
=== Deep Reinforcement Learning ===
Deep
== Applications ==
Deep reinforcement learning has been used for a variety of applications in the past, some of which include:
* The [[AlphaZero]] algorithm, developed by [[DeepMind]], that has achieved super-human like performance in many games.<ref>{{Cite web|title=DeepMind - What if solving one problem could unlock solutions to thousands more?|url=https://deepmind.com/|access-date=2020-11-16|website=Deepmind}}</ref>
* Image enhancement models such as [[Generative adversarial network|GAN]] and Unet which have attained much higher performance compared to the previous methods such as [[Super-resolution imaging|super-resolution]] and segmentation<ref name=":1">{{Cite book
*
== Training ==▼
In order to have a functional agent, the algorithm must be trained with a certain goal. There are different techniques used to train agents, each having their own benefits.▼
[[Q-learning]] attempts to determine the optimal action given a specific state. The way this method determines the Q value, or quality of an action, can be loosely defined by a function taking in a state "s" and and action "a" and outputting the perceived quality of that action:
▲Image enhancement models such as GAN and Unet which have attained much higher performance compared to the previous methods such as [[Super-resolution imaging|super-resolution]] and segmentation<ref>{{Cite book|url=https://www.worldcat.org/oclc/1163522253|title=Deep reinforcement learning fundamentals, research and applications|date=2020|publisher=Springer|others=Dong, Hao., Ding, Zihan., Zhang, Shanghang.|isbn=978-981-15-4095-0|___location=Singapore|oclc=1163522253}}</ref>
<math>Q(s,a)</math>
▲[[Deep Q-learning|Deep Q]] networks, or learning algorithms without a specified model that analyze a situation and produce an action the agent should take.
▲== Training ==
The training process of Q-learning involves exploring different actions and recording the table of q values that correspond state and actions. Once this agent is sufficiently trained, the table should provide an accurate representation of the quality of actions given their state.<ref>{{Cite web|last=Violante|first=Andre|date=2019-07-01|title=Simple Reinforcement Learning: Q-learning|url=https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56|access-date=2020-11-16|website=Medium|language=en}}</ref>
▲In order to have a functional agent, the algorithm must be trained with a certain goal. There are different techniques used to train agents, each having their own benefits.
==== Deep Q-Learning ====
[[Deep Q-learning]] takes the principles of standard Q-learning but approximates the q values using an artificial neural network. In many applications, there is too much input data that needs to be accounted for (e.g. the millions of pixels in a computer screen) which would make the standard process of determining every the q values for each state and action take a large amount of time. By using a neural network to process the data and predict a q value for each available action, the algorithms can be much faster and subsequently, process more data.<ref>{{Cite arXiv|last1=Ong|first1=Hao Yi|last2=Chavez|first2=Kevin|last3=Hong|first3=Augustus|date=2015-10-15|title=Distributed Deep Q-Learning|class=cs.LG |eprint=1508.04186 }}</ref>
=== Challenges ===
There are many factors that cause problems in training using the reinforcement learning method, some of which are listed below:
==== '''Exploration
The exploration exploitation dilemma is the problem of deciding whether to pursue actions that are already known to yield success or explore other actions in order to discover greater success.
In the greedy learning policy the agent chooses actions that have the greatest the q value for the given state:<math>a=argmax_n Q(s,a)</math>With this solution, the agent may get stuck in a local maximum and not discover possible greater success because it only focuses maximizing the q value given its current knowledge.
==== '''Frequency of rewards''' ====▼
In the epsilon-greedy method of training, before determining each action the agent decides whether to prioritize exploration, taking an action with an uncertain outcome for the purpose of gaining more knowledge, or exploitation, picking an action that maximize the q value. At every iteration, a random number between zero and one is selected. If this value is above the specified value of epsilon, the agent will choose a value that prioritizes exploration, otherwise the agent will select an action attempting to maximize the q value. Higher values of epsilon will result in a greater amount of exploration.<ref name=":0">{{Cite journal|last1=Voytenko|first1=S. V.|last2=Galazyuk|first2=A. V.|date=February 2007|title=Intracellular recording reveals temporal integration in inferior colliculus neurons of awake bats|url=https://pubmed.ncbi.nlm.nih.gov/17135472|journal=Journal of Neurophysiology|volume=97|issue=2|pages=1368–1378|doi=10.1152/jn.00976.2006|issn=0022-3077|pmid=17135472}}</ref>
When the goal is too difficult for the learning algorithm to complete, <s>they</s> may never reach the goal and will never be rewarded. Additionally, if a reward is received at the end of a task, the algorithm has no way to differentiate between good and bad behavior during the task. For example, if an algorithm is attempting to learn how to play pong and they make many correct moves but they ultimately loose the point and they are rewarded negatively, there is no way to determine what movements of the paddle were good and what moves were not good due to the reward being too sparse. (<nowiki>https://arxiv.org/abs/2001.00119</nowiki> )▼
==== '''[[Bias–variance tradeoff]]''' ====▼
<math>a = \begin{cases} rand(a_n) & \text{rand(0,1)}\leq\xi \\ argmax_a Q(s,a) & \text{otherwise } \end{cases}</math>
Accounting for the agent's increasing competence over time can be done by using a [[Boltzmann distribution|Boltzmann Distribution]]<nowiki/>learning policy. This works by reducing the amount of exploration over the duration of the training period.
<math>P(a|s) = \frac{e^{(Q(s,a)/T)}}{\Sigma_be^{(Q(s,a)/T)}}</math>
<math>T_{new}=E^{(-dj)}T_{max}+1</math>
Another learning method is [[Simulated annealing|Simulated Annealing.]] In this method, like previously, the agent decides to explore a unknown action or choses an action with the greatest q value based on the equation below:
<math>a = \begin{cases} rand(a_n) & \xi\leq e^{Q(s,rand(a_n)-argmax_aQ(s,a))/T} \\ argmax_a Q(s,a) & \text{otherwise } \end{cases}</math>
In this method as the value T decreases, the more likely the agent is to pursue known beneficial outcomes.<ref name=":0" />
In training reinforcement learning algorithms, agents are rewarded based on their behavior. Variation in the frequency and what occasions that the agent is awarded at can have a large impact on the speed and quality of the outcome of training.
▲When the goal is too difficult for the learning algorithm to complete, <s>they</s> may never reach the goal and will never be rewarded. Additionally, if a reward is received at the end of a task, the algorithm has no way to differentiate between good and bad behavior during the task.<ref>{{Cite journal|last1=Parisi|first1=Simone|last2=Tateo|first2=Davide|last3=Hensel|first3=Maximilian|last4=D'Eramo|first4=Carlo|last5=Peters|first5=Jan|last6=Pajarinen|first6=Joni|date=2019-12-31|title=Long-Term Visitation Value for Deep Exploration in Sparse-Reward Reinforcement Learning|journal=Algorithms |volume=15 |issue=3 |page=81 |doi=10.3390/a15030081 |doi-access=free |arxiv=2001.00119 }}</ref> For example, if an algorithm is attempting to learn how to play pong and they make many correct moves but they ultimately loose the point and they are rewarded negatively, there is no way to determine what movements of the paddle were good and what moves were not good due to the reward being too sparse.
▲==== '''[[Bias–variance tradeoff|Bias–Variance Tradeoff]]''' ====
When training a machine learning model, there is a tradeoff between how well the model fits training data and how well it generalizes to fit the actual data of a problem. This is known as the bias-variance tradeoff. Bias refers to how simple the model is. A high amount of bias will result in a poor fit to most data because it is not able to reflect the complexity of the data. Variance however is how accurately the model fits the training data. A high amount of variance will lead to an overfitting model which will then not be able to be generalized to more data because it will be too specific to the training set of data. This problem means it is important to reduce the bias and variability to find a model that represents the data as simple as possible to be able to generally the data past the training data, but without lacking the complexity of the data.<ref>{{Cite web|last=Singh|first=Seema|date=2018-10-09|title=Understanding the Bias-Variance Tradeoff|url=https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229|access-date=2020-11-16|website=Medium|language=en}}</ref>
=== Optimizations ===
==== '''Reward Shaping''' ====
Reward shaping is the process of giving an agent intermediate rewards
==== '''Curiosity Driven Exploration''' ====
The idea behind curiosity driven exploration is giving the agent a motive to explore unknown outcomes in order to find the best solutions. This is done by "modify[ing] the loss function (or even the network architecture) by adding terms to incentivize exploration"<ref>{{Cite book|last1=Reizinger|first1=Patrik|last2=Szemenyei|first2=Márton|date=2019-10-23|title=ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (
==== '''Hindsight Experience Replay''' ====
Hindsight experience replay is the method of training that involves storing and learning from previous failed attempts to complete a task beyond just a negative reward. While a failed attempt may not have reached the intended goal, it can serve as a lesson for how achieve the unintended result.<ref>{{Cite
== Generalization ==
Reinforcement learning models require an indication state in order to function. When this state is provided by a artificial neural network, which are good at dictating features from raw data (e.g. pixels or raw image files), there is a reduced need to predefine the environment, allowing the model to be generalized to multiple applications. With this layer of abstraction, deep reinforcement learning algorithms can be designed in a way that allows them to become generalized and the same model can be used for different tasks.<ref>{{Cite arXiv|last1=Packer|first1=Charles|last2=Gao|first2=Katelyn|last3=Kos|first3=Jernej|last4=Krähenbühl|first4=Philipp|last5=Koltun|first5=Vladlen|last6=Song|first6=Dawn|date=2019-03-15|title=Assessing Generalization in Deep Reinforcement Learning|class=cs.LG |eprint=1810.12282 }}</ref>
== References ==<!--- See http://en.wikipedia.org/wiki/Wikipedia:Footnotes on how to create references using <ref></ref> tags, these references will then appear here automatically -->
| |||