Content deleted Content added
m typos, minor improvements (to VAE section) |
→Regularized autoencoders: remove abbreviation from section title |
||
Line 51:
Various techniques exist to prevent autoencoders from learning the [[identity function]] and to improve their ability to capture important information and learn richer representations.
====Sparse autoencoder
Inspired by the [[sparse coding]] hypothesis in neuroscience, sparse autoencoders (SAE) are variants of autoencoders, such that the codes <math>E_\phi(x)</math> for messages tend to be ''sparse codes'', that is, <math>E_\phi(x)</math> is close to zero in most entries. Sparse autoencoders may include more (rather than fewer) hidden units than inputs, but only a small number of the hidden units are allowed to be active at the same time.<ref name="domingos" /> Encouraging sparsity improves performance on classification tasks.<ref name=":5">{{Cite journal |last1=Frey |first1=Brendan |last2=Makhzani |first2=Alireza |date=2013-12-19 |title=k-Sparse Autoencoders |arxiv=1312.5663 |bibcode=2013arXiv1312.5663M}}</ref> [[File:Autoencoder sparso.png|thumb|Simple schema of a single-layer sparse autoencoder. The hidden nodes in bright yellow are activated, while the light yellow ones are inactive. The activation depends on the input.]]
There are two main ways to enforce sparsity. One way is to simply clamp all but the highest-k activations of the latent code to zero. This is the '''k-sparse autoencoder'''.<ref name=":1">{{cite arXiv |eprint=1312.5663 |class=cs.LG |first1=Alireza |last1=Makhzani |first2=Brendan |last2=Frey |title=K-Sparse Autoencoders |date=2013}}</ref>
Line 74:
\right]</math>where <math>h_k</math> is the activation vector in the <math>k</math>-th layer of the autoencoder. The norm <math>\|\cdot\|</math> is usually the L1 norm (giving the L1 sparse autoencoder) or the L2 norm (giving the L2 sparse autoencoder).
====Denoising autoencoder
Denoising autoencoders (DAE) try to achieve a ''good'' representation by changing the ''reconstruction criterion''.<ref name=":0" /><ref name=":4" />
| |||