Content deleted Content added
No edit summary |
|||
Line 14:
=== Before modern ===
One origin of RNN was neuroscience. The word "recurrent" is used to describe loop-like structures in [[anatomy]]. In 1901, [[Santiago Ramón y Cajal|Cajal]] observed "recurrent semicircles" in the [[Cerebellum|cerebellar cortex]] formed by [[parallel fiber]], [[Purkinje cell]]s, and [[granule cell]]s.<ref>{{Cite journal |last1=Espinosa-Sanchez |first1=Juan Manuel |last2=Gomez-Marin |first2=Alex |last3=de Castro |first3=Fernando |date=2023-07-05 |title=The Importance of Cajal's and Lorente de Nó's Neuroscience to the Birth of Cybernetics |url=http://journals.sagepub.com/doi/10.1177/10738584231179932 |journal=The Neuroscientist |volume=31 |issue=1 |pages=14–30 |language=en |doi=10.1177/10738584231179932 |pmid=37403768 |hdl=10261/348372 |issn=1073-8584|hdl-access=free }}</ref><ref>{{Cite book |last=Ramón y Cajal |first=Santiago |url=https://archive.org/details/b2129592x_0002/page/n159/mode/2up |title=Histologie du système nerveux de l'homme & des vertébrés |date=1909 |publisher=Paris : A. Maloine |others=Foyle Special Collections Library King's College London |volume=II |pages=149}}</ref> In 1933, [[Rafael Lorente de Nó|Lorente de Nó]] discovered "recurrent, reciprocal connections" by [[Golgi's method]], and proposed that excitatory loops explain certain aspects of the [[vestibulo-ocular reflex]].<ref>{{Cite journal |last=de NÓ |first=R. Lorente |date=1933-08-01 |title=Vestibulo-Ocular Reflex Arc |url=http://archneurpsyc.jamanetwork.com/article.aspx?doi=10.1001/archneurpsyc.1933.02240140009001 |journal=Archives of Neurology and Psychiatry |volume=30 |issue=2 |pages=245 |doi=10.1001/archneurpsyc.1933.02240140009001 |issn=0096-6754|url-access=subscription }}</ref><ref>{{Cite journal |last=Larriva-Sahd |first=Jorge A. |date=2014-12-03 |title=Some predictions of Rafael Lorente de Nó 80 years later |journal=Frontiers in Neuroanatomy |volume=8 |pages=147 |doi=10.3389/fnana.2014.00147 |doi-access=free |issn=1662-5129 |pmc=4253658 |pmid=25520630}}</ref> During 1940s, multiple people proposed the existence of feedback in the brain, which was a contrast to the previous understanding of the neural system as a purely feedforward structure. [[Donald O. Hebb|Hebb]] considered "reverberating circuit" as an explanation for short-term memory.<ref>{{Cite web |title=reverberating circuit |url=https://www.oxfordreference.com/display/10.1093/oi/authority.20110803100417461 |access-date=2024-07-27 |website=Oxford Reference }}</ref> The McCulloch and Pitts paper (1943), which proposed the [[McCulloch-Pitts neuron]] model, considered networks that contains cycles. The current activity of such networks can be affected by activity indefinitely far in the past.<ref>{{Cite journal |last1=McCulloch |first1=Warren S. |last2=Pitts |first2=Walter |date=December 1943 |title=A logical calculus of the ideas immanent in nervous activity |url=http://link.springer.com/10.1007/BF02478259 |journal=The Bulletin of Mathematical Biophysics |volume=5 |issue=4 |pages=115–133 |doi=10.1007/BF02478259 |issn=0007-4985|url-access=subscription }}</ref> They were both interested in closed loops as possible explanations for e.g. [[epilepsy]] and [[Complex regional pain syndrome|causalgia]].<ref>{{Cite journal |last1=Moreno-Díaz |first1=Roberto |last2=Moreno-Díaz |first2=Arminda |date=April 2007 |title=On the legacy of W.S. McCulloch |url=https://linkinghub.elsevier.com/retrieve/pii/S0303264706002152 |journal=Biosystems |volume=88 |issue=3 |pages=185–190 |doi=10.1016/j.biosystems.2006.08.010|pmid=17184902 |bibcode=2007BiSys..88..185M |url-access=subscription }}</ref><ref>{{Cite journal |last=Arbib |first=Michael A |date=December 2000 |title=Warren McCulloch's Search for the Logic of the Nervous System |url=https://muse.jhu.edu/article/46496 |journal=Perspectives in Biology and Medicine |volume=43 |issue=2 |pages=193–216 |doi=10.1353/pbm.2000.0001 |pmid=10804585 |issn=1529-8795|url-access=subscription }}</ref> [[Renshaw cell|Recurrent inhibition]] was proposed in 1946 as a negative feedback mechanism in motor control. Neural feedback loops were a common topic of discussion at the [[Macy conferences]].<ref>{{Cite journal |last=Renshaw |first=Birdsey |date=1946-05-01 |title=Central Effects of Centripetal Impulses in Axons of Spinal Ventral Roots |url=https://www.physiology.org/doi/10.1152/jn.1946.9.3.191 |journal=Journal of Neurophysiology |volume=9 |issue=3 |pages=191–204 |doi=10.1152/jn.1946.9.3.191 |pmid=21028162 |issn=0022-3077|url-access=subscription }}</ref> See <ref name=":0">{{Cite journal |last=Grossberg |first=Stephen |date=2013-02-22 |title=Recurrent Neural Networks |journal=Scholarpedia |volume=8 |issue=2 |pages=1888 |doi=10.4249/scholarpedia.1888 |doi-access=free |bibcode=2013SchpJ...8.1888G |issn=1941-6016}}</ref> for an extensive review of recurrent neural network models in neuroscience.[[File:Typical_connections_in_a_close-loop_cross-coupled_perceptron.png|thumb|A close-loop cross-coupled perceptron network<ref name=":1" />{{Pg|page=403|___location=Fig. 47}}]]
[[Frank Rosenblatt]] in 1960 published "close-loop cross-coupled perceptrons", which are 3-layered [[perceptron]] networks whose middle layer contains recurrent connections that change by a [[Hebbian theory|Hebbian learning]] rule.<ref>F. Rosenblatt, "[[iarchive:SelfOrganizingSystems/page/n87/mode/1up|Perceptual Generalization over Transformation Groups]]", pp. 63--100 in ''Self-organizing Systems: Proceedings of an Inter-disciplinary Conference, 5 and 6 May 1959''. Edited by Marshall C. Yovitz and Scott Cameron. London, New York, [etc.], Pergamon Press, 1960. ix, 322 p.</ref>{{Pg|pages=73-75}} Later, in ''Principles of Neurodynamics'' (1961), he described "closed-loop cross-coupled" and "back-coupled" perceptron networks, and made theoretical and experimental studies for Hebbian learning in these networks,<ref name=":1">{{Cite book |last=Rosenblatt |first=Frank |url=https://archive.org/details/DTIC_AD0256582/page/n3/mode/2up |title=DTIC AD0256582: PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS |date=1961-03-15 |publisher=Defense Technical Information Center |language=english}}</ref>{{Pg|___location=Chapter 19, 21}} and noted that a fully cross-coupled perceptron network is equivalent to an infinitely deep feedforward network.<ref name=":1" />{{Pg|___location=Section 19.11}}
| |||