Content deleted Content added
Olexa Riznyk (talk | contribs) m →Neural networks: Fixing style/layout errors |
Olexa Riznyk (talk | contribs) →Comparison of networks: removing a lost reference |
||
Line 103:
| '''Neuron''' || deterministic binary state. Activation = { 0 (or -1) if x is negative, 1 otherwise } || stochastic binary Hopfield neuron || ← same. (extended to real-valued in mid 2000s) || ← same || ← same || <!--AE--> language: LSTM. vision: local receptive fields. usually real valued relu activation. || middle layer neurons encode means & variances for Gaussians. In run mode (inference), the output of the middle layer are sampled values from the Gaussians.
|-
| '''Connections''' || 1-layer with symmetric weights. No self-connections. || 2-layers. 1-hidden & 1-visible. symmetric weights. || ← same. <br>no lateral connections within a layer. || top layer is undirected, symmetric. other layers are 2-way, asymmetric. || 3-layers: asymmetric weights. 2 networks combined into 1. || <!--AE--> 3-layers. The input is considered a layer even though it has no inbound weights. recurrent layers for NLP. feedforward convolutions for vision. input & output have the same neuron counts. || 3-layers: input, encoder, distribution sampler decoder. the sampler is not considered a layer
|-
| '''Inference & energy''' || Energy is given by Gibbs probability measure :<math>E = -\frac12\sum_{i,j}{w_{ij}{s_i}{s_j}}+\sum_i{\theta_i}{s_i}</math> || ← same || ← same || <!-- --> || minimize KL divergence || inference is only feed-forward. previous UL networks ran forwards AND backwards || minimize error = reconstruction error - KLD
| |||