Sparse distributed memory: Difference between revisions

Revision as of 09:16, 5 July 2021

Revision as of 20:58, 20 August 2021

Line 173:

==Quantum-mechanical interpretation==

[[Quantum superposition]] states that any physical system simultaneously exists in all of its possible [[quantum state|states]], the number of which is exponential in the number of entities composing the system. The strength of presence of each possible state in the superposition{{snd}} i.e., the probability with which it would be observed if measured{{snd}} is represented by its [[probability amplitude]] coefficient. The assumption that these coefficients must be represented physically disjointly from each other, i.e., localisticallylocally, is nearly universal in the [[quantum mechanics|quantum theory]]/[[quantum computing]] literature. Alternatively, as suggested recently by Gerard Rinkus at [[Brandeis University]],<ref name="rinkus12">{{cite journal|doi=10.14704/nq.2012.10.2.507|title=Quantum Computation via Sparse Distributed Representation|journal=NeuroQuantology|volume=10|issue=2|year=2012|last1=Rinkus|first1=Gerard J.|arxiv=1707.05660|s2cid=9754194}}</ref> these coefficients can be represented using sparse distributed representations (SDR) inline with Kanerva's SDM design, wherein each coefficient is represented by a small subset of an overall population of representational units and the subsets can overlap.

Specifically, If we consider an SDR model in which the overall population consists of Q clusters, each having K binary units, so that each coefficient is represented by a set of Q units, one per cluster. We can then consider the particular world state, X, whose coefficient's representation, R(X), is the set of Q units active at time t to have the maximal probability and the probabilities of all other states, Y, to correspond to the size of the intersection of R(Y) and R(X). Thus, R(X) simultaneously serves both as the representation of the particular state, X, and as a probability distribution over all states. When any given code, e.g., R(A), is active, all other codes stored in the model are also physically active in proportion to their intersection with R(A). Thus, SDR provides a classical realization of quantum superposition in which probability amplitudes are represented directly and implicitly by sizes of [[set intersection]]s. If algorithms exist for which the time it takes to store (learn) new representations and to find the closest-matching stored representation ([[Bayesian inference|probabilistic inference]]) remains constant as additional representations are stored, this would meet the criterion of [[quantum computing]].<ref name=rinkus12/> (Also see [[Quantum cognition]] and [[Quantum neural network#Quantum associative memory|Quantum associative memory]])

Line 219:

Negated-translated sigmoid decay mechanism: <math>f(x)=1-[\frac{1}{1+e^{-a(x-c)}}]</math>

In the exponential decay function, it approaches zero more quickly as ''x'' increases, and ''a'' is a constant (usually between 3-9) and ''c'' is a counter. For the negated-[[Translation (geometry)|translated]] [[sigmoid function]], the decay is similar to the exponential decay function when ''a'' is greater than 4.<ref name=memphis />

As the graph approaches 0, it represents how the memory is being forgotten using decay mechanisms.

Line 252:

* Using word vectors of larger size than address vectors: This extension preserves many of the desirable properties of the original SDM: auto-associability, content addressability, distributed storage and robustness over noisy inputs. In addition, it adds new functionality, enabling an efficient auto-associative storage of sequences of vectors, as well as of other data structures such as trees.<ref>{{cite journal | last1 = Snaider | first1 = Javier | last2 = Franklin | first2 = Stan | year = 2012 | title = Extended sparse distributed memory and sequence storage | url = https://www.semanticscholar.org/paper/20298cddb815e5bcbc055415c6a62865c076b3b9| journal = Cognitive Computation | volume = 4 | issue = 2| pages = 172–180 | doi=10.1007/s12559-012-9125-8| s2cid = 14319722 }}</ref>

* Constructing SDM from [[Biological neuron model|Spiking Neurons]]: Despite the biological likeness of SDM most of the work undertaken to demonstrate its capabilities to date has used highly artificial neuron models which abstract away the actual behaviour of [[neurons]] in the [[brain]]. Recent work by [[Steve Furber]]'s lab at the [[University of Manchester]]<ref>{{cite journal | last1 = Furber | first1 = Steve B. |display-authors=etal | year = 2004 | title = Sparse distributed memory using N-of-M codes | journal = Neural Networks | volume = 17 | issue = 10| pages = 1437–1451 | doi=10.1016/j.neunet.2004.07.003| pmid = 15541946 }}</ref><ref>Sharp, Thomas: "[https://studentnet.cs.manchester.ac.uk/resources/library/thesis_abstracts/MSc09/FullText/SharpThomas.pdf Application of sparse distributed memory to the Inverted Pendulum Problem]". Diss. University of Manchester, 2009. URL: http://studentnet.cs.manchester.ac.uk/resources/library/thesis_abstracts/MSc09/FullText/SharpThomas.pdf</ref><ref>Bose, Joy. [https://www.academia.edu/download/7385022/bose07_phd.pdf Engineering a Sequence Machine Through Spiking Neurons Employing Rank-order Codes]. Diss. University of Manchester, 2007.</ref> proposed adaptations to SDM, e.g. by incorporating N-of-M rank codes<ref>Simon Thorpe and Jacques Gautrais. Rank order coding. In Computational Neuroscience: Trends in research, pages 113–118. Plenum Press, 1998.</ref><ref>{{cite journal | last1 = Furber | first1 = Stephen B. |display-authors=etal | year = 2007 | title = Sparse distributed memory using rank-order neural codes | journal = IEEE Transactions on Neural Networks| volume = 18 | issue = 3| pages = 648–659 | doi=10.1109/tnn.2006.890804| pmid = 17526333 | citeseerx = 10.1.1.686.6196 | s2cid = 14256161 }}</ref> into how [[Neural coding#Population coding|populations of neurons]] may encode information—which may make it possible to build an SDM variant from biologically plausible components. This work has been incorporated into [[SpiNNaker|SpiNNaker (Spiking Neural Network Architecture)]] which is being used as the [[Neuromorphic engineering|Neuromorphic Computing]] Platform for the [[Human Brain Project]].<ref>{{cite journal | last1 = Calimera | first1 = A | last2 = Macii | first2 = E | last3 = Poncino | first3 = M | year = 2013 | title = The Human Brain Project and neuromorphic computing | journal = Functional Neurology | volume = 28 | issue = 3| pages = 191–6 | pmid = 24139655 | pmc=3812737}}</ref>

* Non-random distribution of locations:<ref>{{cite journal | last1 = Hely | first1 = Tim | last2 = Willshaw | first2 = David J. | last3 = Hayes | first3 = Gillian M. | year = 1997 | title = A new approach to Kanerva's sparse distributed memory | url = https://semanticscholar.org/paper/2f55ae4083ca073344badc416b83b00fef0db04f| journal = IEEE Transactions on Neural Networks| volume = 8 | issue = 3| pages = 791–794 | doi=10.1109/72.572115| pmid = 18255679 | s2cid = 18628649 }}</ref><ref>Caraig, Lou Marvin. "[https://arxiv.org/abs/1207.5774 A New Training Algorithm for Kanerva's Sparse Distributed Memory]." arXiv preprint arXiv:1207.5774 (2012).</ref> Although the storage locations are initially distributed randomly in the binary N address space, the final distribution of locations depends upon the input patterns presented, and may be non-random thus allowing better flexibility and [[Generalization error|generalization]]. The data pattern is first stored at locations which lie closest to the input address. The signal (i.e. data pattern) then spreads throughout the memory, and a small percentage of the signal strength (e.g. 5%) is lost at each subsequent ___location encountered. Distributing the signal in this way removes the need for a select read/write radius, one of the problematic features of the original SDM. All locations selected in a write operation do not now receive a copy of the original binary pattern with equal strength. Instead they receive a copy of the pattern weighted with a real value from 1.0->0.05 to store in real valued counters (rather than binary counters in Kanerva's SDM). This rewards the nearest locations with a greater signal strength, and uses the natural architecture of the SDM to attenuate the signal strength. Similarly in reading from the memory, output from the nearest locations is given a greater weight than from more distant locations. The new signal method allows the total signal strength received by a ___location to be used as a measure of the fitness of a ___location and is flexible to varying input (as the loss factor does not have to be changed for input patterns of different lengths).

* SDMSCue (Sparse Distributed Memory for Small Cues): Ashraf Anwar & Stan Franklin at The University of Memphis, introduced a variant of SDM capable of Handling Small Cues; namely SDMSCue in 2002. The key idea is to use multiple Reads/Writes, and space projections to reach a successively longer cue.<ref>{{Cite book|title = A Sparse Distributed Memory Capable of Handling Small Cues, SDMSCue|publisher = Springer US|date = 2005-01-01|isbn = 978-0-387-24048-0|pages = 23–38|series = IFIP — The International Federation for Information Processing|language = en|first1 = Ashraf|last1 = Anwar|first2 = Stan|last2 = Franklin|editor-first = Michael K.|editor-last = Ng|editor-first2 = Andrei|editor-last2 = Doncescu|editor-first3 = Laurence T.|editor-last3 = Yang|editor-first4 = Tau|editor-last4 = Leng|doi = 10.1007/0-387-24049-7_2}}</ref>