Recurrent neural network: Difference between revisions

Revision as of 01:54, 9 February 2025

Revision as of 04:50, 9 February 2025

Line 3:

{{Machine learning|Artificial neural network}}

'''Recurrent neural networks''' ('''RNNs)''') are a class of [[Neural network (machine learning)|artificial neural network]]networks commonly useddesigned for sequential data processing. Unlike [[feedforward neural network]]s, which processsequential data in a single pass, RNNssuch process data across multiple time steps, making them well-adapted for modelling and processingas text, speech, and [[time series]].[[time series|,]]<ref>{{Cite journal |last1=Tealab |first1=Ahmed |date=2018-12-01 |title=Time series forecasting using artificial neural networks methodologies: A systematic review |journal=Future Computing and Informatics Journal |volume=3 |issue=2 |pages=334–340 |doi=10.1016/j.fcij.2018.10.003 |issn=2314-7288 |doi-access=free}}</ref> where the order of elements is important. Unlike [[feedforward neural network]]s, which process inputs independently, RNNs utilize recurrent connections, where the output of a neuron at one time step is fed back as input to the network at the next time step. This enables RNNs to capture temporal dependencies and patterns within sequences.

The fundamental building block of RNNs is the '''recurrent unit''', which maintains a '''hidden state'''—a form of memory that is updated at each time step based on the current input and the previous hidden state. This feedback mechanism allows the network to learn from past inputs and incorporate that knowledge into its current processing. RNNs have been successfully applied to tasks such as unsegmented, connected [[handwriting recognition]],<ref>{{cite journal |last1=Graves |first1=Alex |author-link1=Alex Graves (computer scientist) |last2=Liwicki |first2=Marcus |last3=Fernandez |first3=Santiago |last4=Bertolami |first4=Roman |last5=Bunke |first5=Horst |last6=Schmidhuber |first6=Jürgen |author-link6=Jürgen Schmidhuber |year=2009 |title=A Novel Connectionist System for Improved Unconstrained Handwriting Recognition |url=http://www.idsia.ch/~juergen/tpami_2008.pdf |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=31 |issue=5 |pages=855–868 |yearciteseerx=200910.1.1.139.4502 |doi=10.1109/tpami.2008.137 |pmid=19299860 |citeseerx=10.1.1.139.4502 |s2cid=14635907 }}</ref> [[speech recognition]],<ref name="sak2014">{{Cite web |urllast1=https://research.google.com/pubs/archive/43905.pdfSak |publisherfirst1=GoogleHaşim Research|last2=Senior |first2=Andrew |last3=Beaufays |first3=Françoise |year=2014 |title=Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling |last1url=Sakhttps://research.google.com/pubs/archive/43905.pdf |first1publisher=Haşim |last2=Senior |first2=Andrew |last3=Beaufays | first3=Françoise |year=2014Google Research}}</ref><ref name="liwu2015">{{cite arXiv |last1eprint=Li1410.4281 |class=cs.CL |first1=Xiangang |last2last1=WuLi |first2=Xihong |datelast2=2014-10-15Wu |title=Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition |eprintdate=1410.4281 |class=cs.CL 2014-10-15}}</ref> [[natural language processing]], and [[neural machine translation]].<ref>{{Cite journal |last=Dupond |first=Samuel |date=2019 |title=<!-- for sure correct title? not found, nor in archive.org (for 2020-02-13), nor Volume correct? 2019 is vol 47-48 and 41 from 2016--> A thorough review on the current advance of neural network structures. |url=https://www.sciencedirect.com/journal/annual-reviews-in-control |journal=Annual Reviews in Control |volume=14 |pages=200–230}}</ref><ref>{{Cite journal |last1=Abiodun |first1=Oludare Isaac |last2=Jantan |first2=Aman |last3=Omolara |first3=Abiodun Esther |last4=Dada |first4=Kemi Victoria |last5=Mohamed |first5=Nachaat Abdelatif |last6=Arshad |first6=Humaira |date=2018-11-01 |title=State-of-the-art in artificial neural network applications: A survey |journal=Heliyon |volume=4 |issue=11 |pages=e00938 |bibcode=2018Heliy...400938A |doi=10.1016/j.heliyon.2018.e00938 |issn=2405-8440 |pmc=6260436 |pmid=30519653 |doi-access=free}}</ref>

The building block of RNNs is the '''recurrent unit'''. This unit maintains a hidden state, essentially a form of memory, which is updated at each time step based on the current input and the previous hidden state. This feedback loop allows the network to learn from past inputs, and incorporate that knowledge into its current processing.

EarlyHowever, traditional RNNs sufferedsuffer from the [[vanishing gradient problem]], limitingwhich limits their ability to learn long-range dependencies. This issue was solvedaddressed by the development of the [[long short-term memory]] (LSTM) variantarchitecture in 1997, thus making it the standard architectureRNN variant for RNNhandling long-term dependencies. Later, [[Gated recurrent unit|Gated Recurrent Units]] (GRUs) were introduced as a more computationally efficient alternative.

In recent years, [[Transformer (deep learning architecture)|Transformers]], which rely on self-attention mechanisms instead of recurrence, have become the dominant architecture for many sequence-processing tasks, particularly in natural language processing, due to their superior handling of long-range dependencies and greater parallelizability. Nevertheless, RNNs remain relevant for applications where computational efficiency, real-time processing, or the inherent sequential nature of data is crucial.

RNNs have been applied to tasks such as unsegmented, connected [[handwriting recognition]],<ref>{{cite journal |last1=Graves |first1=Alex |author-link1=Alex Graves (computer scientist) |last2=Liwicki |first2=Marcus |last3=Fernandez |first3=Santiago |last4=Bertolami |first4=Roman |last5=Bunke |first5=Horst |last6=Schmidhuber |first6=Jürgen |author-link6=Jürgen Schmidhuber |title=A Novel Connectionist System for Improved Unconstrained Handwriting Recognition |url=http://www.idsia.ch/~juergen/tpami_2008.pdf |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=31 |issue=5 |pages=855–868 |year=2009 |doi=10.1109/tpami.2008.137 |pmid=19299860 |citeseerx=10.1.1.139.4502 |s2cid=14635907 }}</ref> [[speech recognition]],<ref name="sak2014">{{Cite web |url=https://research.google.com/pubs/archive/43905.pdf |publisher=Google Research |title=Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling |last1=Sak |first1=Haşim |last2=Senior |first2=Andrew |last3=Beaufays | first3=Françoise |year=2014 }}</ref><ref name="liwu2015">{{cite arXiv |last1=Li |first1=Xiangang |last2=Wu |first2=Xihong |date=2014-10-15 |title=Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition |eprint=1410.4281 |class=cs.CL }}</ref> [[natural language processing]], and [[neural machine translation]].<ref>{{Cite journal |last=Dupond |first=Samuel |date=2019 |title=<!-- for sure correct title? not found, nor in archive.org (for 2020-02-13), nor Volume correct? 2019 is vol 47-48 and 41 from 2016--> A thorough review on the current advance of neural network structures. |url=https://www.sciencedirect.com/journal/annual-reviews-in-control |journal=Annual Reviews in Control |volume=14 |pages=200–230}}</ref><ref>{{Cite journal |last1=Abiodun |first1=Oludare Isaac |last2=Jantan |first2=Aman |last3=Omolara |first3=Abiodun Esther |last4=Dada |first4=Kemi Victoria |last5=Mohamed |first5=Nachaat Abdelatif |last6=Arshad |first6=Humaira |date=2018-11-01 |title=State-of-the-art in artificial neural network applications: A survey |journal=Heliyon |volume=4 |issue=11 |pages=e00938 |bibcode=2018Heliy...400938A |doi=10.1016/j.heliyon.2018.e00938 |issn=2405-8440 |pmc=6260436 |pmid=30519653 |doi-access=free}}</ref>

{{toclimit|3}}

==History==