Content deleted Content added
Line 171:
Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal|first1=Martin|last1=Rehn|first2=Friedrich T.|last2=Sommer|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields|journal=Journal of Computational Neuroscience|year=2007|volume=22|issue=2|pages=135–146|doi=10.1007/s10827-006-0003-9|pmid=17053994|s2cid=294586|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination|linear fashion]] to approximate the input.
More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis (linear algebra)|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math>, corresponding to neuronal receptive fields, along with a [[Sparse vector|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal|last1=Lee|first1=Honglak|last2=Battle|first2=Alexis|last3=Raina|first3=Rajat|last4=Ng|first4=Andrew Y.|title=Efficient sparse coding algorithms|journal=Advances in Neural Information Processing Systems|year=2006|url=https://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>
The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>
| |||