Content deleted Content added
| (20 intermediate revisions by 4 users not shown) | |||
Line 1:
{{Short description|Class of artificial neural network}}
{{Distinguish|Recursive neural network|Feedback neural network}}
{{Machine learning|Neural networks}}
Line 30:
[[Long short-term memory]] (LSTM) networks were invented by [[Sepp Hochreiter|Hochreiter]] and [[Jürgen Schmidhuber|Schmidhuber]] in 1995 and set accuracy records in multiple applications domains.<ref>{{Cite Q|Q98967430}}</ref><ref name="lstm">{{Cite journal |last1=Hochreiter |first1=Sepp |author-link=Sepp Hochreiter |last2=Schmidhuber |first2=Jürgen |date=1997-11-01 |title=Long Short-Term Memory |journal=Neural Computation |volume=9 |issue=8 |pages=1735–1780 |doi=10.1162/neco.1997.9.8.1735|pmid=9377276 |s2cid=1915014 }}</ref> It became the default choice for RNN architecture.
[[Bidirectional recurrent neural networks]] (BRNN)
Around 2006, bidirectional LSTM started to revolutionize [[speech recognition]], outperforming traditional models in certain speech applications.<ref>{{Cite journal |last1=Graves |first1=Alex |last2=Schmidhuber |first2=Jürgen |date=2005-07-01 |title=Framewise phoneme classification with bidirectional LSTM and other neural network architectures |journal=Neural Networks |series=IJCNN 2005 |volume=18 |issue=5 |pages=602–610 |citeseerx=10.1.1.331.5800 |doi=10.1016/j.neunet.2005.06.042 |pmid=16112549 |s2cid=1856462}}</ref><ref name="fernandez2007keyword">{{Cite conference |last1=Fernández |first1=Santiago |last2=Graves |first2=Alex |last3=Schmidhuber |first3=Jürgen |year=2007 |title=An Application of Recurrent Neural Networks to Discriminative Keyword Spotting |url=http://dl.acm.org/citation.cfm?id=1778066.1778092 |book-title=Proceedings of the 17th International Conference on Artificial Neural Networks |series=ICANN'07 |___location=Berlin, Heidelberg |publisher=Springer-Verlag |pages=220–229 |isbn=978-3-540-74693-5 }}</ref> They also improved large-vocabulary speech recognition<ref name="sak2014" /><ref name="liwu2015" /> and [[text-to-speech]] synthesis<ref name="fan2015">{{cite conference |last1=Fan |first1=Bo |last2=Wang |first2=Lijuan |last3=Soong |first3=Frank K. |last4=Xie |first4=Lei |title=Photo-Real Talking Head with Deep Bidirectional LSTM |chapter-url= |editor= |book-title=Proceedings of ICASSP 2015 IEEE International Conference on Acoustics, Speech and Signal Processing |doi=10.1109/ICASSP.2015.7178899 |date=2015 |isbn=978-1-4673-6997-8 |pages=4884–8 }}</ref> and was used in [[Google Voice Search|Google voice search]], and dictation on [[Android (operating system)|Android devices]].<ref name="sak2015">{{Cite web |url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html |title=Google voice search: faster and more accurate |last1=Sak |first1=Haşim |last2=Senior |first2=Andrew |date=September 2015 |last3=Rao |first3=Kanishka |last4=Beaufays |first4=Françoise |last5=Schalkwyk |first5=Johan}}</ref> They broke records for improved [[machine translation]],<ref name="sutskever2014">{{Cite journal |last1=Sutskever |first1=Ilya |last2=Vinyals |first2=Oriol |last3=Le |first3=Quoc V. |year=2014 |title=Sequence to Sequence Learning with Neural Networks |url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf |journal=Electronic Proceedings of the Neural Information Processing Systems Conference |volume=27 |page=5346 |arxiv=1409.3215 |bibcode=2014arXiv1409.3215S }}</ref> [[Language Modeling|language modeling]]<ref name="vinyals2016">{{cite arXiv |last1=Jozefowicz |first1=Rafal |last2=Vinyals |first2=Oriol |last3=Schuster |first3=Mike |last4=Shazeer |first4=Noam |last5=Wu |first5=Yonghui |date=2016-02-07 |title=Exploring the Limits of Language Modeling |eprint=1602.02410 |class=cs.CL}}</ref> and Multilingual Language Processing.<ref name="gillick2015">{{cite arXiv |last1=Gillick |first1=Dan |last2=Brunk |first2=Cliff |last3=Vinyals |first3=Oriol |last4=Subramanya |first4=Amarnag |date=2015-11-30 |title=Multilingual Language Processing From Bytes |eprint=1512.00103 |class=cs.CL}}</ref> Also, LSTM combined with [[convolutional neural network]]s (CNNs) improved [[automatic image captioning]].<ref name="vinyals2015">{{cite arXiv |last1=Vinyals |first1=Oriol |last2=Toshev |first2=Alexander |last3=Bengio |first3=Samy |last4=Erhan |first4=Dumitru |date=2014-11-17 |title=Show and Tell: A Neural Image Caption Generator |eprint=1411.4555 |class=cs.CV }}</ref>
Line 38:
==Configurations==
{{main|Layer (deep learning)}}
An RNN-based model can be factored into two parts: configuration and architecture. Multiple RNNs can be combined in a data flow, and the data flow itself is the configuration. Each RNN itself may have any architecture, including LSTM, GRU, etc.
Line 159 ⟶ 160:
{{Main|Bidirectional associative memory}}
Introduced by [[Bart Kosko]],<ref>{{cite journal |year=1988 |title=Bidirectional associative memories |journal=IEEE Transactions on Systems, Man, and Cybernetics |volume=18 |issue=1 |pages=49–60 |doi=10.1109/21.87054 |last1=Kosko |first1=Bart |s2cid=59875735 }}</ref> a bidirectional associative memory (BAM) network is a variant of a Hopfield network that stores associative data as a vector. The bidirectionality comes from passing information through a matrix and its [[transpose]]. Typically, [[bipolar encoding]] is preferred to binary encoding of the associative pairs. Recently, stochastic BAM models using [[Markov chain|Markov]] stepping
A BAM network has two layers, either of which can be driven as an input to recall an association and produce an output on the other layer.<ref>{{cite book
Line 179 ⟶ 180:
{{Main|Recursive neural network}}
A '''[[recursive neural network]]'''<ref>{{cite book |last1=Goller |first1=Christoph |title=Proceedings of International Conference on Neural Networks (ICNN'96) |last2=Küchler |first2=Andreas |year=1996 |isbn=978-0-7803-3210-2 |volume=1 |page=347 |chapter=Learning task-dependent distributed representations by backpropagation through structure |citeseerx=10.1.1.52.4759 |doi=10.1109/ICNN.1996.548916 |s2cid=6536466}}</ref> is created by applying the same set of weights [[recursion|recursively]] over a differentiable graph-like structure by traversing the structure in [[topological sort|topological order]]. Such networks are typically also trained by the reverse mode of [[automatic differentiation]].<ref name="lin1970">{{cite thesis |first=Seppo |last=Linnainmaa |author-link=Seppo Linnainmaa |year=1970 |title=The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors |type=MSc |language=fi |publisher=University of Helsinki}}</ref><ref name="grie2008">{{cite book |last1=Griewank |first1=Andreas |url={{google books |plainurl=y |id=xoiiLaRxcbEC}} |title=Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation |last2=Walther |first2=Andrea |author2-link=Andrea Walther |publisher=SIAM |year=2008 |isbn=978-0-89871-776-1 |edition=Second}}</ref> They can process [[distributed representation]]s of structure, such as [[mathematical logic|logical terms]]. A special case of recursive neural networks is the RNN whose structure corresponds to a linear chain. Recursive neural networks have been applied to [[natural language processing]].<ref>{{citation |last1=Socher |first1=Richard |title=28th International Conference on Machine Learning (ICML 2011) |contribution=Parsing Natural Scenes and Natural Language with Recursive Neural Networks |contribution-url=https://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf |last2=Lin |first2=Cliff |last3=Ng |first3=Andrew Y. |last4=Manning |first4=Christopher D.}}</ref> The
===Neural Turing machines===
Line 186 ⟶ 187:
'''Neural Turing machines''' (NTMs) are a method of extending recurrent neural networks by coupling them to external [[memory]] resources with which they interact. The combined system is analogous to a [[Turing machine]] or [[Von Neumann architecture]] but is [[Differentiable neural computer|differentiable]] end-to-end, allowing it to be efficiently trained with [[gradient descent]].<ref>{{cite arXiv |eprint=1410.5401 |class=cs.NE |first1=Alex |last1=Graves |first2=Greg |last2=Wayne |title=Neural Turing Machines |last3=Danihelka |first3=Ivo |year=2014}}</ref>
'''Differentiable neural computers''' (DNCs) are an extension of
'''Neural network pushdown automata''' (NNPDA) are similar to NTMs, but tapes are replaced by analog stacks that are differentiable and trained. In this way, they are similar in complexity to recognizers of [[context free grammar]]s (CFGs).<ref>{{Cite book |last1=Sun |first1=Guo-Zheng |title=Adaptive Processing of Sequences and Data Structures |last2=Giles |first2=C. Lee |last3=Chen |first3=Hsing-Hen |publisher=Springer |year=1998 |isbn=978-3-540-64341-8 |editor-last=Giles |editor-first=C. Lee |series=Lecture Notes in Computer Science |___location=Berlin, Heidelberg |pages=296–345 |chapter=The Neural Network Pushdown Automaton: Architecture, Dynamics and Training |citeseerx=10.1.1.56.8723 |doi=10.1007/bfb0054003 |editor-last2=Gori |editor-first2=Marco}}</ref>
Recurrent neural networks are [[Turing complete]] and can run arbitrary programs to process arbitrary sequences of inputs.<ref>{{cite journal |last1=Hyötyniemi |first1=Heikki |date=1996 |title=Turing machines are recurrent neural networks |journal=Proceedings of STeP '96/Publications of the Finnish Artificial Intelligence Society |pages=13–24}}</ref>
Line 208 ⟶ 209:
Gradient descent is a [[:Category:First order methods|first-order]] [[Iterative algorithm|iterative]] [[Mathematical optimization|optimization]] [[algorithm]] for finding the minimum of a function. In neural networks, it can be used to minimize the error term by changing each weight in proportion to the derivative of the error with respect to that weight, provided the non-linear [[activation function]]s are [[Differentiable function|differentiable]].
{{anchor|Real-Time Recurrent Learning}}The standard method for training RNN by gradient descent is the "[[backpropagation through time]]" (BPTT) algorithm, which is a special case of the general algorithm of [[backpropagation]]. A more computationally expensive online variant is called "Real-Time Recurrent Learning" or RTRL,<ref>{{cite book |last1=Robinson |first1=Anthony J.<!-- sometimes cited as T. (for "Tony") Robinson --> |url={{google books |plainurl=y |id=6JYYMwEACAAJ }} |title=The Utility Driven Dynamic Error Propagation Network |last2=Fallside |first2=Frank |publisher=Department of Engineering, University of Cambridge |year=1987 |series=Technical Report CUED/F-INFENG/TR.1}}</ref><ref>{{cite book |last1=Williams |first1=Ronald J. |url={{google books |plainurl=y |id=B71nu3LDpREC}} |title=Backpropagation: Theory, Architectures, and Applications |last2=Zipser |first2=D. |date=1 February 2013 |publisher=Psychology Press |isbn=978-1-134-77581-1 |editor-last1=Chauvin |editor-first1=Yves |contribution=Gradient-based learning algorithms for recurrent networks and their computational complexity |editor-last2=Rumelhart |editor-first2=David E.}}</ref> which is an instance of [[automatic differentiation]] in the forward accumulation mode with stacked tangent vectors. Unlike BPTT, this algorithm is [[Local algorithm|local]] in time but not local in space.
In this context, local in space means that a unit's weight vector can be updated using only information stored in the connected units and the unit itself such that update complexity of a single unit is linear in the dimensionality of the weight vector. Local in time means that the updates take place continually (on-line) and depend only on the most recent time step rather than on multiple time steps within a given time horizon as in BPTT. Biological neural networks appear to be local with respect to both time and space.<ref>{{Cite journal |last=Schmidhuber |first=Jürgen |date=1989-01-01 |title=A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks |journal=Connection Science |volume=1 |issue=4 |pages=403–412 |doi=10.1080/09540098908915650 |s2cid=18721007}}</ref><ref name="PríncipeEuliano2000">{{cite book |last1=Príncipe |first1=José C. |url={{google books |plainurl=y |id=jgMZAQAAIAAJ}} |title=Neural and adaptive systems: fundamentals through simulations |last2=Euliano |first2=Neil R. |last3=Lefebvre |first3=W. Curt |publisher=Wiley |year=2000 |isbn=978-0-471-35167-2}}</ref>
Line 214 ⟶ 215:
For recursively computing the partial derivatives, RTRL has a time-complexity of O(number of hidden x number of weights) per time step for computing the [[Jacobian matrix|Jacobian matrices]], while BPTT only takes O(number of weights) per time step, at the cost of storing all forward activations within the given time horizon.<ref name="Ollivier2015">{{Cite arXiv |eprint=1507.07680 |class=cs.NE |first1=Ollivier |last1=Yann |first2=Corentin |last2=Tallec |title=Training recurrent networks online without backtracking |date=2015-07-28 |first3=Guillaume |last3=Charpiat}}</ref> An online hybrid between BPTT and RTRL with intermediate complexity exists,<ref>{{Cite journal |last=Schmidhuber |first=Jürgen |date=1992-03-01 |title=A Fixed Size Storage O(n3) Time Complexity Learning Algorithm for Fully Recurrent Continually Running Networks |journal=Neural Computation |volume=4 |issue=2 |pages=243–248 |doi=10.1162/neco.1992.4.2.243 |s2cid=11761172}}</ref><ref>{{cite report |url=http://citeseerx.ist.psu.edu/showciting?cid=128036 |title=Complexity of exact gradient computation algorithms for recurrent neural networks |last=Williams |first=Ronald J. |publisher=Northeastern University, College of Computer Science |___location=Boston (MA) |access-date=2017-07-02 |archive-url=https://web.archive.org/web/20171020033840/http://citeseerx.ist.psu.edu/showciting?cid=128036 |archive-date=2017-10-20 |url-status=dead |series=Technical Report NU-CCS-89-27 |year=1989}}</ref> along with variants for continuous time.<ref>{{Cite journal |last=Pearlmutter |first=Barak A. |date=1989-06-01 |title=Learning State Space Trajectories in Recurrent Neural Networks |url=http://repository.cmu.edu/cgi/viewcontent.cgi?article=2865&context=compsci |journal=Neural Computation |volume=1 |issue=2 |pages=263–269 |doi=10.1162/neco.1989.1.2.263 |s2cid=16813485}}</ref>
A major problem with gradient descent for standard RNN architectures is that [[Vanishing gradient problem|error gradients vanish]] exponentially quickly with the size of the time lag between important events.<ref name="hochreiter1991" /><ref name="HOCH2001">{{cite book |last=Hochreiter |first=Sepp |title=A Field Guide to Dynamical Recurrent Networks |date=15 January 2001 |publisher=John Wiley & Sons |isbn=978-0-7803-5369-5 |editor-last1=Kolen |editor-first1=John F. |chapter=Gradient flow in recurrent nets: the difficulty of learning long-term dependencies |display-authors=etal |editor-last2=Kremer |editor-first2=Stefan C. |chapter-url={{google books |plainurl=y |id=NWOcMVA64aAC }}}}</ref> LSTM combined with a BPTT/RTRL hybrid learning method attempts to overcome these problems.<ref name="lstm" /> This problem is also solved in the independently recurrent neural network (IndRNN)<ref name="auto" /> by reducing the context of a neuron to its own past state and the cross-neuron information can then be explored in the following layers. Memories of different ranges including long-term memory can be learned without the gradient vanishing and exploding
The
One approach to gradient information computation in RNNs with arbitrary architectures is based on signal-flow graphs diagrammatic derivation.<ref>{{Cite journal |last1=Wan |first1=Eric A. |last2=Beaufays |first2=Françoise |year=1996 |title=Diagrammatic derivation of gradient algorithms for neural networks |journal=Neural Computation |volume=8 |pages=182–201 |doi=10.1162/neco.1996.8.1.182 |s2cid=15512077}}</ref> It uses the BPTT batch algorithm, based on Lee's theorem for network sensitivity calculations.<ref name="ReferenceA">{{Cite journal |last1=Campolucci |first1=Paolo |last2=Uncini |first2=Aurelio |last3=Piazza |first3=Francesco |year=2000 |title=A Signal-Flow-Graph Approach to On-line Gradient Calculation |journal=Neural Computation |volume=12 |issue=8 |pages=1901–1927 |citeseerx=10.1.1.212.5406 |doi=10.1162/089976600300015196 |pmid=10953244 |s2cid=15090951}}</ref> It was proposed by Wan and Beaufays, while its fast online version was proposed by Campolucci, Uncini and Piazza.<ref name="ReferenceA" />
Line 234 ⟶ 235:
* The mean-squared error is returned to the fitness function.
* This function drives the genetic selection process.
Many chromosomes make up the population; therefore, many different neural networks are evolved until a stopping criterion is satisfied. A common stopping scheme
* When the neural network has learned a certain percentage of the training data
* When the minimum value of the mean-squared-error is satisfied
* When the maximum number of training generations has been reached.
The fitness function evaluates the stopping criterion as it receives the mean-squared error reciprocal from each network during training. Therefore, the goal of the genetic algorithm is to maximize the fitness function, reducing the mean-squared error.
Line 245 ⟶ 246:
===Independently RNN (IndRNN) ===
The independently recurrent neural network (IndRNN)<ref name="auto">{{cite arXiv |title= Independently Recurrent Neural Network (IndRNN): Building a Longer and Deeper RNN|last1=Li |first1=Shuai |last2=Li |first2=Wanqing |last3=Cook |first3=Chris |last4=Zhu |first4=Ce |last5=Yanbo |first5=Gao |eprint=1803.04831|class=cs.CV |year=2018 }}</ref> addresses the gradient vanishing and exploding problems in the traditional fully connected RNN. Each neuron in one layer only receives its own past state as context information (instead of full connectivity to all other neurons in this layer) and thus neurons are independent of each other's history. The gradient backpropagation can be regulated to avoid gradient vanishing and exploding in order to keep long or short-term memory. The cross-neuron information is explored in the next layers. IndRNN can be robustly trained with non-saturated nonlinear functions such as [[ReLU]]. Deep networks can be trained using [[skip connections]].
===Neural history compressor===
The ''neural history compressor'' is an unsupervised stack of RNNs.<ref name="schmidhuber1992">{{cite journal |last1=Schmidhuber |first1=Jürgen |year=1992 |title=Learning complex, extended sequences using the principle of history compression |url=ftp://ftp.idsia.ch/pub/juergen/chunker.pdf |journal=Neural Computation |volume=4 |issue=2 |pages=234–242 |doi=10.1162/neco.1992.4.2.234 |archive-url=https://web.archive.org/web/20170706014739/ftp://ftp.idsia.ch/pub/juergen/chunker.pdf |archive-date=2017-07-06 |url-status=dead |s2cid=18271205 }}</ref> At the input level, it learns to predict its next input from the previous inputs. Only unpredictable inputs of some RNN in the hierarchy become inputs to the next higher level RNN, which therefore recomputes its internal state only rarely. Each higher level RNN thus studies a compressed representation of the information in the RNN below. This is done such that the input sequence can be precisely reconstructed from the representation at the highest level.
The system effectively minimizes the description length or the negative [[logarithm]] of the probability of the data.<ref name="scholarpedia2015pre">{{cite journal |last1=Schmidhuber |first1=Jürgen |year=2015 |title=Deep Learning |journal=Scholarpedia |volume=10 |issue=11 |page=32832 |doi=10.4249/scholarpedia.32832 |bibcode=2015SchpJ..1032832S |doi-access=free }}</ref> Given a lot of learnable predictability in the incoming data sequence, the highest level RNN can use supervised learning to easily classify even deep sequences with long intervals between important events.
Line 257 ⟶ 259:
===Second order RNNs===
Second-order RNNs use higher order weights <math>w{}_{ijk}</math> instead of the standard <math>w{}_{ij}</math> weights, and states can be a product. This allows a direct mapping to a [[finite-state machine]] both in training
===Hierarchical recurrent neural network===
Line 268 ⟶ 270:
===Multiple timescales model===
A multiple timescales recurrent neural network (MTRNN) is a neural-based computational model that can simulate the functional hierarchy of the brain through self-organization depending on the spatial connection between neurons and on distinct types of neuron activities, each with distinct time properties.<ref>{{Cite journal |last1=Yamashita |first1=Yuichi |last2=Tani |first2=Jun |date=2008-11-07 |title=Emergence of Functional Hierarchy in a Multiple Timescale Neural Network Model: A Humanoid Robot Experiment |journal=PLOS Computational Biology |volume=4 |issue=11 |pages=e1000220 |doi=10.1371/journal.pcbi.1000220 |pmc=2570613 |pmid=18989398 |bibcode=2008PLSCB...4E0220Y |doi-access=free }}</ref><ref>{{Cite journal |last1=Alnajjar |first1=Fady |last2=Yamashita |first2=Yuichi |last3=Tani |first3=Jun |year=2013 |title=The hierarchical and functional connectivity of higher-order cognitive mechanisms: neurorobotic model to investigate the stability and flexibility of working memory |journal=Frontiers in Neurorobotics |volume=7 |page=2 |doi=10.3389/fnbot.2013.00002 |pmc=3575058 |pmid=23423881|doi-access=free }}</ref> With such varied neuronal activities, continuous sequences of any set of behaviors are segmented into reusable primitives, which in turn are flexibly integrated into diverse sequential behaviors. The biological approval of such a type of hierarchy was discussed in the [[memory-prediction framework|memory-prediction]] theory of brain function by [[Jeff Hawkins|Hawkins]] in his book ''[[On Intelligence]]''.{{Citation needed |date=June 2017}} Such a hierarchy also agrees with theories of memory posited by philosopher [[Henri Bergson]], which have been incorporated into an MTRNN model.<ref name="auto1"/><ref>{{Cite web | url=http://jnns.org/conference/2018/JNNS2018_Technical_Programs.pdf | title=
===Memristive networks===
Line 286 ⟶ 288:
}}</ref> The [[memristors]] (memory resistors) are implemented by thin film materials in which the resistance is electrically tuned via the transport of ions or oxygen vacancies within the film. [[DARPA]]'s [[SyNAPSE|SyNAPSE project]] has funded IBM Research and HP Labs, in collaboration with the Boston University Department of Cognitive and Neural Systems (CNS), to develop neuromorphic architectures that may be based on memristive systems.
[[Memristive networks]] are a particular type of [[physical neural network]] that have very similar properties to (Little-)Hopfield networks, as they have continuous dynamics, a limited memory capacity and natural relaxation via the minimization of a function which is asymptotic to the [[Ising model]]. In this sense, the dynamics of a memristive circuit have the advantage compared to a Resistor-Capacitor network to have a more interesting non-linear behavior. From this point of view, engineering analog memristive networks account for a peculiar type of [[neuromorphic engineering]] in which the device behavior depends on the circuit wiring or topology.
The evolution of these networks can be studied analytically using variations of the [[
=== Continuous-time ===
Line 305 ⟶ 307:
CTRNNs have been applied to [[evolutionary robotics]] where they have been used to address vision,<ref>{{citation |last1=Harvey |first1=Inman |title=3rd international conference on Simulation of adaptive behavior: from animals to animats 3 |pages=392–401 |year=1994 |contribution=Seeing the light: Artificial evolution, real vision |contribution-url=https://www.researchgate.net/publication/229091538_Seeing_the_Light_Artificial_Evolution_Real_Vision |last2=Husbands |first2=Phil |last3=Cliff |first3=Dave}}</ref> co-operation,<ref name="Evolving communication without dedicated communication channels">{{cite conference |last=Quinn |first=Matt |year=2001 |title=Evolving communication without dedicated communication channels |pages=357–366 |doi=10.1007/3-540-44811-X_38 |isbn=978-3-540-42567-0 |book-title=Advances in Artificial Life: 6th European Conference, ECAL 2001}}</ref> and minimal cognitive behaviour.<ref name="The dynamics of adaptive behavior: A research program">{{cite journal |last=Beer |first=Randall D. |year=1997 |title=The dynamics of adaptive behavior: A research program |journal=Robotics and Autonomous Systems |volume=20 |issue=2–4 |pages=257–289 |doi=10.1016/S0921-8890(96)00063-2}}</ref>
Note that, by the [[Shannon sampling theorem]], discrete-time recurrent neural networks can be viewed as continuous-time recurrent neural networks where the differential equations have transformed into equivalent [[difference equation]]s.<ref name="Sherstinsky-NeurIPS2018-CRACT-3">{{cite conference |last=Sherstinsky |first=Alex |date=2018-12-07 |editor-last=Bloem-Reddy |editor-first=Benjamin |editor2-last=Paige |editor2-first=Brooks |editor3-last=Kusner |editor3-first=Matt |editor4-last=Caruana |editor4-first=Rich |editor5-last=Rainforth |editor5-first=Tom |editor6-last=Teh |editor6-first=Yee Whye |title=Deriving the Recurrent Neural Network Definition and RNN Unrolling Using Signal Processing |url=https://www.researchgate.net/publication/331718291 |conference=Critiquing and Correcting Trends in Machine Learning Workshop at NeurIPS-2018 |conference-url=https://ml-critique-correct.github.io/}}</ref> This transformation can be thought of as occurring after the post-synaptic node activation functions <math>y_i(t)</math> have been [[Low-pass filter|low-pass filtered]] but prior to sampling.
They are in fact [[recursive neural network]]s with a particular structure: that of a linear chain. Whereas recursive neural networks operate on any hierarchical structure, combining child representations into parent representations, recurrent neural networks operate on the linear progression of time, combining the previous time step and a hidden representation into the representation for the current time step.
From a time-series perspective, RNNs can appear as nonlinear versions of [[finite impulse response]] and [[infinite impulse response]] filters and also as a [[nonlinear autoregressive exogenous model]] (NARX).<ref>{{cite journal |url={{google books |plainurl=y |id=830-HAAACAAJ |page=208}} |title=Computational Capabilities of Recurrent NARX Neural Networks |last1=Siegelmann |first1=Hava T. |last2=Horne |first2=Bill G. |last3=Giles |first3=C. Lee |journal= IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics|volume=27 |issue=2 |pages=208–15 |year=1995 |pmid=18255858 |doi=10.1109/3477.558801 |citeseerx=10.1.1.48.7468 }}</ref> RNN has infinite impulse response whereas [[convolutional neural network]]
The effect of memory-based learning for the recognition of sequences can also be implemented by a more biological-based model which uses the silencing mechanism exhibited in neurons with a relatively high frequency [[Action potential|spiking activity]].<ref>{{Cite journal |last1=Hodassman |first1=Shiri |last2=Meir |first2=Yuval |last3=Kisos |first3=Karin |last4=Ben-Noam |first4=Itamar |last5=Tugendhaft |first5=Yael |last6=Goldental |first6=Amir |last7=Vardi |first7=Roni |last8=Kanter |first8=Ido |date=2022-09-29 |title=Brain inspired neuronal silencing mechanism to enable reliable sequence identification |journal=Scientific Reports |volume=12 |issue=1 |pages=16003 |doi=10.1038/s41598-022-20337-x |pmid=36175466 |pmc=9523036 |arxiv=2203.13028 |bibcode=2022NatSR..1216003H |issn=2045-2322|doi-access=free }}</ref>
Additional stored states and the storage under direct control by the network can be added to both [[infinite impulse response|infinite-impulse]] and [[finite impulse response|finite-impulse]] networks. Another network or graph can also replace the storage if that incorporates time delays or has feedback loops. Such controlled states are referred to as gated states or gated memory and are part of [[long short-term memory]] networks (LSTMs) and [[gated recurrent unit]]s. This is also called Feedback Neural Network (FNN).
| |||