Content deleted Content added
→Neuroscience: a bit more detailed information on Hopfield network |
Thumperward (talk | contribs) fully two thirds of this article is academic trivia; move the bit that isn't up a bit |
||
Line 202:
{{main|Lee–Yang theorem}}
After Onsager's solution, Yang and Lee investigated the way in which the partition function becomes singular as the temperature approaches the critical temperature.
==Applications==
===Magnetism===
The original motivation for the model was the phenomenon of [[ferromagnetism]]. Iron is magnetic; once it is magnetized it stays magnetized for a long time compared to any atomic time.
In the 19th century, it was thought that magnetic fields are due to currents in matter, and [[André-Marie Ampère|Ampère]] postulated that permanent magnets are caused by permanent atomic currents. The motion of classical charged particles could not explain permanent currents though, as shown by [[Joseph Larmor|Larmor]]. In order to have ferromagnetism, the atoms must have permanent [[magnetic moment]]s which are not due to the motion of classical charges.
Once the electron's spin was discovered, it was clear that the magnetism should be due to a large number of electron spins all oriented in the same direction. It was natural to ask how the electrons' spins all know which direction to point in, because the electrons on one side of a magnet don't directly interact with the electrons on the other side. They can only influence their neighbors. The Ising model was designed to investigate whether a large fraction of the electron spins could be oriented in the same direction using only local forces.
===Lattice gas===
The Ising model can be reinterpreted as a statistical model for the motion of atoms. Since the kinetic energy depends only on momentum and not on position, while the statistics of the positions only depends on the potential energy, the thermodynamics of the gas only depends on the potential energy for each configuration of atoms.
A coarse model is to make space-time a lattice and imagine that each position either contains an atom or it doesn't. The space of configuration is that of independent bits ''B<sub>i</sub>'', where each bit is either 0 or 1 depending on whether the position is occupied or not. An attractive interaction reduces the energy of two nearby atoms. If the attraction is only between nearest neighbors, the energy is reduced by −4''JB''<sub>''i''</sub>''B''<sub>''j''</sub> for each occupied neighboring pair.
The density of the atoms can be controlled by adding a [[chemical potential]], which is a multiplicative probability cost for adding one more atom. A multiplicative factor in probability can be reinterpreted as an additive term in the logarithm – the energy. The extra energy of a configuration with ''N'' atoms is changed by ''μN''. The probability cost of one more atom is a factor of exp(−''βμ'').
So the energy of the lattice gas is:
<math display="block">E = - \frac{1}{2} \sum_{\langle i,j \rangle} 4 J B_i B_j + \sum_i \mu B_i</math>
Rewriting the bits in terms of spins, <math>B_i = (S_i + 1)/2. </math>
<math display="block">E = - \frac{1}{2} \sum_{\langle i,j \rangle} J S_i S_j - \frac{1}{2} \sum_i (4 J - \mu) S_i</math>
For lattices where every site has an equal number of neighbors, this is the Ising model with a magnetic field ''h'' = (''zJ'' − ''μ'')/2, where ''z'' is the number of neighbors.
In biological systems, modified versions of the lattice gas model have been used to understand a range of binding behaviors. These include the binding of ligands to receptors in the cell surface,<ref>{{Cite journal|last1=Shi|first1=Y.|last2=Duke|first2=T.|date=1998-11-01|title=Cooperative model of bacteril sensing|journal=Physical Review E|language=en|volume=58|issue=5|pages=6399–6406|doi=10.1103/PhysRevE.58.6399|arxiv=physics/9901052|bibcode=1998PhRvE..58.6399S|s2cid=18854281}}</ref> the binding of [[chemotaxis]] proteins to the flagellar motor,<ref>{{Cite journal|last1=Bai|first1=Fan|last2=Branch|first2=Richard W.|last3=Nicolau|first3=Dan V.|last4=Pilizota|first4=Teuta|last5=Steel|first5=Bradley C.|last6=Maini|first6=Philip K.|last7=Berry|first7=Richard M.|date=2010-02-05|title=Conformational Spread as a Mechanism for Cooperativity in the Bacterial Flagellar Switch|journal=Science|language=en|volume=327|issue=5966|pages=685–689|doi=10.1126/science.1182105|issn=0036-8075|pmid=20133571|bibcode = 2010Sci...327..685B |s2cid=206523521}}</ref> and the condensation of DNA.<ref>{{Cite journal|last1=Vtyurina|first1=Natalia N.|last2=Dulin|first2=David|last3=Docter|first3=Margreet W.|last4=Meyer|first4=Anne S.|last5=Dekker|first5=Nynke H.|last6=Abbondanzieri|first6=Elio A.|date=2016-04-18|title=Hysteresis in DNA compaction by Dps is described by an Ising model|journal=Proceedings of the National Academy of Sciences|language=en|pages=4982–7|doi=10.1073/pnas.1521241113|issn=0027-8424|pmid=27091987|pmc=4983820|volume=113|issue=18|bibcode=2016PNAS..113.4982V|doi-access=free}}</ref>
===Neuroscience===
The activity of [[neuron]]s in the brain can be modelled statistically. Each neuron at any time is either active + or inactive −. The active neurons are those that send an [[action potential]] down the axon in any given time window, and the inactive ones are those that do not.
Following the general approach of Jaynes,<ref>{{Citation| author=Jaynes, E. T.| title= Information Theory and Statistical Mechanics | journal= Physical Review| volume = 106 | pages= 620–630 | year= 1957| doi=10.1103/PhysRev.106.620| postscript=.|bibcode = 1957PhRv..106..620J| issue=4 | s2cid= 17870175 }}</ref><ref>{{Citation| author= Jaynes, Edwin T.| title = Information Theory and Statistical Mechanics II |journal = Physical Review |volume =108 | pages = 171–190 | year = 1957| doi= 10.1103/PhysRev.108.171| postscript= .|bibcode = 1957PhRv..108..171J| issue= 2 }}</ref> a later interpretation of Schneidman, Berry, Segev and Bialek,<ref>{{Citation|author1=Elad Schneidman |author2=Michael J. Berry |author3=Ronen Segev |author4=William Bialek | title= Weak pairwise correlations imply strongly correlated network states in a neural population| journal=Nature| volume= 440 | pages= 1007–1012| year=2006| doi= 10.1038/nature04701| pmid= 16625187| issue= 7087| pmc= 1785327| postscript= .|arxiv = q-bio/0512013 |bibcode = 2006Natur.440.1007S |title-link=neural population }}</ref>
is that the Ising model is useful for any model of neural function, because a statistical model for neural activity should be chosen using the [[principle of maximum entropy]]. Given a collection of neurons, a statistical model which can reproduce the average firing rate for each neuron introduces a [[Lagrange multiplier]] for each neuron:
<math display="block">E = - \sum_i h_i S_i</math>
But the activity of each neuron in this model is statistically independent. To allow for pair correlations, when one neuron tends to fire (or not to fire) along with another, introduce pair-wise lagrange multipliers:
<math display="block">E= - \tfrac{1}{2} \sum_{ij} J_{ij} S_i S_j - \sum_i h_i S_i</math>
where <math>J_{ij}</math> are not restricted to neighbors. Note that this generalization of Ising model is sometimes called the quadratic exponential binary distribution in statistics.
This energy function only introduces probability biases for a spin having a value and for a pair of spins having the same value. Higher order correlations are unconstrained by the multipliers. An activity pattern sampled from this distribution requires the largest number of bits to store in a computer, in the most efficient coding scheme imaginable, as compared with any other distribution with the same average activity and pairwise correlations. This means that Ising models are relevant to any system which is described by bits which are as random as possible, with constraints on the pairwise correlations and the average number of 1s, which frequently occurs in both the physical and social sciences.
===Spin glasses===
With the Ising model the so-called [[spin glasses]] can also be described, by the usual Hamiltonian <math display="inline">H=-\frac{1}{2}\,\sum J_{i,k}\,S_i\,S_k,</math> where the ''S''-variables describe the Ising spins, while the ''J<sub>i,k</sub>'' are taken from a random distribution. For spin glasses a typical distribution chooses antiferromagnetic bonds with probability ''p'' and ferromagnetic bonds with probability 1 − ''p'' (also known as the random-bond Ising model). These bonds stay fixed or "quenched" even in the presence of thermal fluctuations. When ''p'' = 0 we have the original Ising model. This system deserves interest in its own; particularly one has "non-ergodic" properties leading to strange relaxation behaviour. Much attention has been also attracted by the related bond and site dilute Ising model, especially in two dimensions, leading to intriguing critical behavior.<ref>{{Citation|author= J-S Wang, [[Walter Selke|W Selke]], VB Andreichenko, and VS Dotsenko| title= The critical behaviour of the two-dimensional dilute model|journal= Physica A|volume= 164| issue= 2| pages= 221–239 |year= 1990|doi=10.1016/0378-4371(90)90196-Y|bibcode = 1990PhyA..164..221W }}</ref>
=== Artificial neural network ===
{{Main|Hopfield network}}
Ising model was instrumental in the development of the [[Hopfield network]]. The original Ising model is a model for equilibrium. [[Roy J. Glauber]] in 1963 studied the Ising model evolving in time, as a process towards thermal equilibrium ([[Glauber dynamics]]), adding in the component of time.<ref name=":222">{{cite journal |last1=Glauber |first1=Roy J. |date=February 1963 |title=Roy J. Glauber "Time-Dependent Statistics of the Ising Model" |url=https://aip.scitation.org/doi/abs/10.1063/1.1703954 |journal=Journal of Mathematical Physics |volume=4 |issue=2 |pages=294–307 |doi=10.1063/1.1703954 |access-date=2021-03-21}}</ref> (Kaoru Nakano, 1971)<ref name="Nakano1971">{{cite book |last1=Nakano |first1=Kaoru |title=Pattern Recognition and Machine Learning |date=1971 |isbn=978-1-4615-7568-9 |pages=172–186 |chapter=Learning Process in a Model of Associative Memory |doi=10.1007/978-1-4615-7566-5_15}}</ref><ref name="Nakano1972">{{cite journal |last1=Nakano |first1=Kaoru |date=1972 |title=Associatron-A Model of Associative Memory |journal=IEEE Transactions on Systems, Man, and Cybernetics |volume=SMC-2 |issue=3 |pages=380–388 |doi=10.1109/TSMC.1972.4309133}}</ref> and ([[Shun'ichi Amari]], 1972),<ref name="Amari19722">{{cite journal |last1=Amari |first1=Shun-Ichi |date=1972 |title=Learning patterns and pattern sequences by self-organizing nets of threshold elements |journal=IEEE Transactions |volume=C |issue=21 |pages=1197–1206}}</ref> proposed to modify the weights of an Ising model by [[Hebbian theory|Hebbian learning]] rule as a model of associative memory. The same idea was published by ({{ill|William A. Little (physicist)|lt=William A. Little|de|William A. Little}}, 1974),<ref name="little74">{{cite journal |last=Little |first=W. A. |year=1974 |title=The Existence of Persistent States in the Brain |journal=Mathematical Biosciences |volume=19 |issue=1–2 |pages=101–120 |doi=10.1016/0025-5564(74)90031-5}}</ref> who was cited by Hopfield in his 1982 paper.
The [[Spin glass#Sherrington–Kirkpatrick model|Sherrington–Kirkpatrick model]] of spin glass, published in 1975,<ref>{{Cite journal |last1=Sherrington |first1=David |last2=Kirkpatrick |first2=Scott |date=1975-12-29 |title=Solvable Model of a Spin-Glass |url=https://link.aps.org/doi/10.1103/PhysRevLett.35.1792 |journal=Physical Review Letters |volume=35 |issue=26 |pages=1792–1796 |bibcode=1975PhRvL..35.1792S |doi=10.1103/PhysRevLett.35.1792 |issn=0031-9007}}</ref> is the Hopfield network with random initialization. Sherrington and Kirkpatrick found that it is highly likely for the energy function of the SK model to have many local minima. In the 1982 paper, Hopfield applied this recently developed theory to study the Hopfield network with binary activation functions.<ref name="Hopfield1982">{{cite journal |last1=Hopfield |first1=J. J. |date=1982 |title=Neural networks and physical systems with emergent collective computational abilities |journal=Proceedings of the National Academy of Sciences |volume=79 |issue=8 |pages=2554–2558 |bibcode=1982PNAS...79.2554H |doi=10.1073/pnas.79.8.2554 |pmc=346238 |pmid=6953413 |doi-access=free}}</ref> In a 1984 paper he extended this to continuous activation functions.<ref name=":03">{{cite journal |last1=Hopfield |first1=J. J. |date=1984 |title=Neurons with graded response have collective computational properties like those of two-state neurons |journal=Proceedings of the National Academy of Sciences |volume=81 |issue=10 |pages=3088–3092 |bibcode=1984PNAS...81.3088H |doi=10.1073/pnas.81.10.3088 |pmc=345226 |pmid=6587342 |doi-access=free}}</ref> It became a standard model for the study of neural networks through statistical mechanics.<ref>{{Cite book |last1=Engel |first1=A. |title=Statistical mechanics of learning |last2=Broeck |first2=C. van den |date=2001 |publisher=Cambridge University Press |isbn=978-0-521-77307-2 |___location=Cambridge, UK ; New York, NY}}</ref><ref>{{Cite journal |last1=Seung |first1=H. S. |last2=Sompolinsky |first2=H. |last3=Tishby |first3=N. |date=1992-04-01 |title=Statistical mechanics of learning from examples |url=https://journals.aps.org/pra/abstract/10.1103/PhysRevA.45.6056 |journal=Physical Review A |volume=45 |issue=8 |pages=6056–6091 |bibcode=1992PhRvA..45.6056S |doi=10.1103/PhysRevA.45.6056 |pmid=9907706}}</ref>
===Sea ice===
The [[melt pond]] can be modelled by the Ising model; sea ice topography data bears rather heavily on the results. The state variable is binary for a simple 2D approximation, being either water or ice.<ref>{{cite arXiv|author= Yi-Ping Ma|author2= Ivan Sudakov|author3= Courtenay Strong|author4= Kenneth Golden|title= Ising model for melt ponds on Arctic sea ice|year= 2017|class= physics.ao-ph|eprint=1408.2487v3}}</ref>
===Cayley tree topologies and large neural networks===
[[File:Cayley Tree Branch with Branching Ratio = 2.jpg|thumb|An Open Cayley Tree or Branch with Branching Ratio = 2 and k Generations]]
In order to investigate an Ising model with potential relevance for large (e.g. with <math>10^4</math> or <math>10^5</math> interactions per node) neural nets, at the suggestion of Krizan in 1979, {{harvtxt|Barth|1981}} obtained the exact analytical expression for the free energy of the Ising model on the closed Cayley tree (with an arbitrarily large branching ratio) for a zero-external magnetic field (in the thermodynamic limit) by applying the methodologies of {{harvtxt|Glasser|1970}} and {{harvtxt|Jellito|1979}}
<math display="block">-\beta f = \ln 2 + \frac{2\gamma}{(\gamma+1)} \ln (\cosh J) + \frac{\gamma(\gamma-1)}{(\gamma+1)} \sum_{i=2}^z\frac{1}{\gamma^i}\ln J_i (\tau) </math>
[[File:Closed Cayley Tree with Branching Ratio = 4.jpg |thumb|A Closed Cayley Tree with Branching Ratio = 4. (Only sites for generations k, k-1, and k=1(overlapping as one row) are shown for the joined trees)]] where <math>\gamma</math> is an arbitrary branching ratio (greater than or equal to 2), <math>t \equiv \tanh J</math>, <math>\tau \equiv t^2</math>, <math>J \equiv \beta\epsilon</math> (with <math>\epsilon</math> representing the nearest-neighbor interaction energy) and there are k (→ ∞ in the thermodynamic limit) generations in each of the tree branches (forming the closed tree architecture as shown in the given closed Cayley tree diagram.) The sum in the last term can be shown to converge uniformly and rapidly (i.e. for z → ∞, it remains finite) yielding a continuous and monotonous function, establishing that, for <math>\gamma</math> greater than or equal to 2, the free energy is a continuous function of temperature T. Further analysis of the free energy indicates that it exhibits an unusual discontinuous first derivative at the critical temperature ({{harvtxt|Krizan|Barth|Glasser|1983}}, {{harvtxt|Glasser|Goldberg|1983}}.)
The spin-spin correlation between sites (in general, m and n) on the tree was found to have a transition point when considered at the vertices (e.g. A and Ā, its reflection), their respective neighboring sites (such as B and its reflection), and between sites adjacent to the top and bottom extreme vertices of the two trees (e.g. A and B), as may be determined from
<math display="block">\langle s_m s_n \rangle = {Z_N}^{-1}(0,T) [\cosh J]^{N_b} 2^N \sum_{l=1}^z g_{mn}(l) t^l</math>
where <math>N_b</math> is equal to the number of bonds, <math>g_{mn}(l)t^l</math> is the number of graphs counted for odd vertices with even intermediate sites (see cited methodologies and references for detailed calculations), <math>2^N</math> is the multiplicity resulting from two-valued spin possibilities and the partition function <math>{Z_N}</math> is derived from <math>\sum_{\{s\}}e^{-\beta H}</math>. (Note: <math>s_i </math> is consistent with the referenced literature in this section and is equivalent to <math>S_i</math> or <math>\sigma_i</math> utilized above and in earlier sections; it is valued at <math>\pm 1 </math>.) The critical temperature <math>T_C</math> is given by
<math display="block">T_C = \frac{2\epsilon}{k_\text{B}[\ln(\sqrt \gamma+1) - \ln(\sqrt \gamma-1)]}.</math>
The critical temperature for this model is only determined by the branching ratio <math>\gamma</math> and the site-to-site interaction energy <math>\epsilon</math>, a fact which may have direct implications associated with neural structure vs. its function (in that it relates the energies of interaction and branching ratio to its transitional behavior.) For example, a relationship between the transition behavior of activities of neural networks between sleeping and wakeful states (which may correlate with a spin-spin type of phase transition) in terms of changes in neural interconnectivity (<math>\gamma</math>) and/or neighbor-to-neighbor interactions (<math>\epsilon</math>), over time, is just one possible avenue suggested for further experimental investigation into such a phenomenon. In any case, for this Ising model it was established, that “the stability of the long-range correlation increases with increasing <math>\gamma</math> or increasing <math>\epsilon</math>.”
For this topology, the spin-spin correlation was found to be zero between the extreme vertices and the central sites at which the two trees (or branches) are joined (i.e. between A and individually C, D, or E.) This behavior is explained to be due to the fact that, as k increases, the number of links increases exponentially (between the extreme vertices) and so even though the contribution to spin correlations decrease exponentially, the correlation between sites such as the extreme vertex (A) in one tree and the extreme vertex in the joined tree (Ā) remains finite (above the critical temperature.) In addition, A and B also exhibit a non-vanishing correlation (as do their reflections) thus lending itself to, for B level sites (with A level), being considered “clusters” which tend to exhibit synchronization of firing.
Based upon a review of other classical network models as a comparison, the Ising model on a closed Cayley tree was determined to be the first classical statistical mechanical model to demonstrate both local and long-range sites with non-vanishing spin-spin correlations, while at the same time exhibiting intermediate sites with zero correlation, which indeed was a relevant matter for large neural networks at the time of its consideration. The model's behavior is also of relevance for any other divergent-convergent tree physical (or biological) system exhibiting a closed Cayley tree topology with an Ising-type of interaction. This topology should not be ignored since its behavior for Ising models has been solved exactly, and presumably nature will have found a way of taking advantage of such simple symmetries at many levels of its designs.
{{harvtxt|Barth|1981}} early on noted the possibility of interrelationships between (1) the classical large neural network model (with similar coupled divergent-convergent topologies) with (2) an underlying statistical quantum mechanical model (independent of topology and with persistence in fundamental quantum states):
{{Blockquote|The most significant result obtained from the closed Cayley tree model involves the occurrence of long-range correlation in the absence of intermediate-range correlation. This result has not been demonstrated by other classical models. The failure of the classical view of impulse transmission to account for this phenomenon has been cited by numerous investigators (Ricciiardi and Umezawa, 1967, Hokkyo 1972, Stuart, Takahashi and Umezawa 1978, 1979) as significant enough to warrant radically new assumptions on a very fundamental level and have suggested the existence of quantum cooperative modes within the brain…In addition, it is interesting to note that the (modeling) of…Goldstone particles or bosons (as per Umezawa, et al)…within the brain, demonstrates the long-range correlation of quantum numbers preserved in the ground state…In the closed Cayley tree model ground states of pairs of sites, as well as the state variable of individual sites, (can) exhibit long-range correlation.|author=|title=|source=}}
It was a natural and common belief among early neurophysicists (e.g. Umezawa, Krizan, Barth, etc.) that classical neural models (including those with statistical mechanical aspects) will one day have to be integrated with quantum physics (with quantum statistical aspects), similar perhaps to how the ___domain of chemistry has historically integrated itself into quantum physics via quantum chemistry.
Several additional statistical mechanical problems of interest remain to be solved for the closed Cayley tree, including the time-dependent case and the external field situation, as well as theoretical efforts aimed at understanding interrelationships with underlying quantum constituents and their physics.
==Monte Carlo methods for numerical simulation==
Line 792 ⟶ 871:
Variants of this method produce good numerical approximations for the critical exponents when many terms are included, in both two and three dimensions.
==See also==
| |||