The [[Hopfield network|'''[[Hopfield network]]''']] is an RNN in which all connections across layers are equally sized. It requires [[Stationary process|stationary]] inputs and is thus not a general RNN, as it does not process sequences of patterns. However, it guarantees that it will converge. If the connections are trained using [[Hebbian learning]], then the Hopfield network can perform as [[Robustness (computer science)|robust]] [[content-addressable memory]], resistant to connection alteration.
==={{Anchor|Elman network|Jordan network}}Elman networks and Jordan networks===
Line 181:
{{Main|Recursive neural network}}
A [[recursive neural network|'''[[recursive neural network]]''']]<ref>{{cite book |last1=Goller |first1=Christoph |title=Proceedings of International Conference on Neural Networks (ICNN'96) |last2=Küchler |first2=Andreas |year=1996 |isbn=978-0-7803-3210-2 |volume=1 |page=347 |chapter=Learning task-dependent distributed representations by backpropagation through structure |citeseerx=10.1.1.52.4759 |doi=10.1109/ICNN.1996.548916 |s2cid=6536466}}</ref> is created by applying the same set of weights [[recursion|recursively]] over a differentiable graph-like structure by traversing the structure in [[topological sort|topological order]]. Such networks are typically also trained by the reverse mode of [[automatic differentiation]].<ref name="lin1970">{{cite thesis |first=Seppo |last=Linnainmaa |author-link=Seppo Linnainmaa |year=1970 |title=The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors |type=MSc |language=fi |publisher=University of Helsinki}}</ref><ref name="grie2008">{{cite book |last1=Griewank |first1=Andreas |url={{google books |plainurl=y |id=xoiiLaRxcbEC}} |title=Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation |last2=Walther |first2=Andrea |author2-link=Andrea Walther |publisher=SIAM |year=2008 |isbn=978-0-89871-776-1 |edition=Second}}</ref> They can process [[distributed representation]]s of structure, such as [[mathematical logic|logical terms]]. A special case of recursive neural networks is the RNN whose structure corresponds to a linear chain. Recursive neural networks have been applied to [[natural language processing]].<ref>{{citation |last1=Socher |first1=Richard |title=28th International Conference on Machine Learning (ICML 2011) |contribution=Parsing Natural Scenes and Natural Language with Recursive Neural Networks |contribution-url=https://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf |last2=Lin |first2=Cliff |last3=Ng |first3=Andrew Y. |last4=Manning |first4=Christopher D.}}</ref> The Recursive Neural Tensor Network uses a [[tensor]]-based composition function for all nodes in the tree.<ref>{{cite journal |last1=Socher |first1=Richard |last2=Perelygin |first2=Alex |last3=Wu |first3=Jean Y. |last4=Chuang |first4=Jason |last5=Manning |first5=Christopher D. |last6=Ng |first6=Andrew Y. |last7=Potts |first7=Christopher |title=Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank |url=http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf |journal=Emnlp 2013}}</ref>
