Content deleted Content added
MichaelMaggs (talk | contribs) Adding short description: "Type of artificial neural network" |
|||
| (11 intermediate revisions by 8 users not shown) | |||
Line 1:
{{Short description|Type of artificial neural network}}
A ''' The idea is to add structures called
Among other benefits, capsnets address the "Picasso problem" in image recognition: images that have all the right parts but that are not in the correct spatial relationship (e.g., in a "face", the positions of the mouth and one eye are switched). For image recognition, capsnets exploit the fact that while viewpoint changes have nonlinear effects at the pixel level, they have linear effects at the part/object level.<ref name=":16">{{cite web|url=http://www.cedar.buffalo.edu/~srihari/CSE676/9.12%20CapsuleNets.pdf|title=Capsule Nets|last=Srihari|first=Sargur|publisher=[[University of Buffalo]]|access-date=2017-12-07}}</ref> This can be compared to inverting the rendering of an object of multiple parts.<ref name=":0">{{Cite book|url=http://papers.nips.cc/paper/1710-learning-to-parse-images.pdf|title=Advances in Neural Information Processing Systems 12|last1=Hinton|first1=Geoffrey E|last2=Ghahramani|first2=Zoubin|last3=Teh|first3=Yee Whye|date=2000|publisher=MIT Press|editor-last=Solla|editor-first=S. A.|editor-link=Sara Solla|pages=463–469|editor-last2=Leen|editor-first2=T. K.|editor-last3=Müller|editor-first3=K.}}</ref>
Line 30 ⟶ 31:
== Pooling ==
Capsnets reject the [[
* violates biological shape perception in that it has no intrinsic coordinate frame;
* provides invariance (discarding positional information) instead of equivariance (disentangling that information);
Line 62 ⟶ 63:
The pose vector <math display="inline">\mathbf{u}_{i}</math> is rotated and translated by a matrix <math display="inline">\mathbf{W}_{ij}</math> into a vector <math display="inline">\mathbf{\hat{u}}_{j|i}</math> that predicts the output of the parent capsule.
Capsules <math display="inline">s_{j}</math> in the next higher level are fed the sum of the predictions from all capsules in the lower layer, each with a coupling coefficient <math display="inline">c_{ij}</math>
==== Procedure softmax ====
The coupling coefficients from a capsule <math display="inline">i</math> in layer <math display="inline">l</math> to all capsules in layer <math display="inline">l+1</math> sum to one, and are defined by a "[[Softmax function|routing softmax]]". The initial [[logit]]s <math display="inline">b_{ij}</math> are prior [[Log probability|log probabilities]] for the routing. That is the [[prior probability]] that capsule <math display="inline">i</math> in layer <math display="inline">l</math> should connect to capsule <math display="inline">j</math> in layer <math display="inline">l+1</math>. Normalization of the coupling coefficients:<ref name=":1"/>
1: \mathbf{procedure}~ \mathrm{softmax} ( \mathbf{b}, i ) \\
2: \quad \triangleright \mbox{argument matrix} \\
Line 85 ⟶ 86:
==== Procedure squash ====
Because the length of the vectors represents probabilities they should be between zero
1: \mathbf{procedure}~ \mathrm{squash} ( \mathbf{a} ) \\
2: \quad \triangleright \mbox{argument vector} \\
Line 100 ⟶ 101:
One approach to routing is the following<ref name=":1"/>
~~1: \mathbf{procedure}~ \mathrm{routing} ( \mathbf{\hat{u}}_{j|i}, r, l ) \\
~~2: \quad \triangleright \mbox{argument vector} \\
Line 132 ⟶ 133:
=== Margin loss ===
The length of the instantiation vector represents the probability that a capsule's entity is present in the scene. A top-level capsule has a long vector if and only if its associated entity is present. To allow for multiple entities, a separate [[Hinge loss|margin loss]] is computed for each capsule. Downweighting the loss for absent entities stops the learning from shrinking activity vector lengths for all entities. The total loss is the sum of the losses of all entities.<ref name=":1"/> In Hinton's example the loss function is:<ref name=":1"/>
L_{k} & = \underbrace{T_{k} ~ { \max \left ( 0, m^{+} - \| \mathbf{v}_{k} \| \right )}^{2}}_\mbox{class present}
+ \underbrace{\lambda \left ( 1 - T_{k} \right ) ~ { \max \left ( 0, \| \mathbf{v}_{k} \| - m^{-} \right )}^{2}}_\mbox{class not present}
Line 156 ⟶ 157:
Capsnets are hierarchical, in that each lower-level capsule contributes significantly to only one higher-level capsule.<ref name=":1"/>
However, replicating learned knowledge remains valuable. To achieve this, a capsnet's lower layers are [[convolution]]al, including hidden capsule layers. Higher layers thus cover larger regions, while retaining information about the precise position of each object within the region. For low level capsules, ___location information is
== Human vision ==
Human vision examines a sequence of focal points (directed by [[saccade]]s), processing only a fraction of the scene at its highest resolution. Capsnets build on inspirations from [[cortical minicolumn]]s (also called cortical microcolumns) in the [[cerebral cortex]]. A minicolumn is a structure containing 80-120 neurons, with a diameter of about 28-40
Capsnets explore the intuition that the human visual system creates a [[Parse tree|tree]]-like structure for each focal point and coordinates these trees to recognize objects. However, with capsnets each tree is "carved" from a fixed network (by adjusting coefficients) rather than assembled on the fly.<ref name=":1"/>
Line 188 ⟶ 189:
==References==
{{reflist|2|refs=
<ref name=":1">{{Cite
}}
== External links ==
* {{Citation|title=Capsules Network Implementation in PyTorch, fixing several bugs in previous implementations |date=2018-04-16|url=https://github.com/manuelsh/capsule-networks-pytorch|access-date=2018-04-16}}
* {{Citation|title=Pytorch code: Capsule Routing via Variational Bayes | date=February 2020|url=https://github.com/fabio-deep/Variational-Capsule-Routing|access-date=2020-10-23}}
* {{Citation|title=A PyTorch implementation of the NIPS 2017 paper "Dynamic Routing Between Capsules"|date=2017-12-08|url=https://github.com/gram-ai/capsule-networks|publisher=Gram.AI|access-date=2017-12-08}}
* {{
* {{Cite web|url=http://www.cedar.buffalo.edu/~srihari/CSE676|title=Deep Learning|website=www.cedar.buffalo.edu|access-date=2017-12-07}}
*{{Cite web|url=https://medium.freecodecamp.org/understanding-capsule-networks-ais-alluring-new-architecture-bdb228173ddc|title=Understanding Capsule Networks — AI's Alluring New Architecture|last=Bourdakos|first=Nick|date=2018-02-12|website=freeCodeCamp.org|access-date=2019-04-23}}
*{{Cite
*{{Cite
* {{Citation|last=Guo|first=Xifeng|title=CapsNet-Keras: A Keras implementation of CapsNet in NIPS2017 paper "Dynamic Routing Between Capsules". Now test error ＝ 0.34%.|date=2017-12-08|url=https://github.com/XifengGuo/CapsNet-Keras|access-date=2017-12-08}}
* {{Cite web|url=https://openreview.net/pdf?id=HJWLfGWRb|title=MATRIX CAPSULES WITH EM ROUTING|last1=Hinton|first1=Geoffrey|last2=Sabour|first2=Sara|last3=Frosst|first3=Nicholas|date=November 2017}}
* {{
* {{Citation|last=Liao|first=Huadong|title=CapsNet-Tensorflow: A Tensorflow implementation of CapsNet(Capsules Net) in Hinton's paper Dynamic Routing Between Capsules|date=2017-12-08|url=https://github.com/naturomics/CapsNet-Tensorflow|access-date=2017-12-08}}
*{{Cite web|first=Fangyu|last=Cai|date=2020-12-18|title='We Can Do It' — Geoffrey Hinton and UBC, UT, Google & UVic Team Propose Unsupervised
* {{cite
[[Category:Artificial neural networks]]
| |||