Mathematics of neural networks in machine learning: Difference between revisions

Revision as of 01:51, 3 September 2019

Revision as of 02:46, 4 September 2019

Line 38:

Mathematically, a neuron's network function <math>\textstyle f(x)</math> is defined as a composition of other functions <math>\textstyle g_i(x)</math>, that can further be decomposed into other functions. This can be conveniently represented as a network structure, with arrows depicting the dependencies between functions. A widely used type of composition is the ''nonlinear weighted sum'', where <math>\textstyle f (x) = K \left(\sum_i w_i g_i(x)\right) </math>, where <math>\textstyle K</math> (commonly referred to as the [[activation function]]<ref>{{Cite web|url=http://www.cse.unsw.edu.au/~billw/mldict.html#activnfn|title=The Machine Learning Dictionary|website=www.cse.unsw.edu.au}}</ref>) is some predefined function, such as the [[Hyperbolic function#Standard analytic expressions|hyperbolic tangent]], [[sigmoid function]], [[softmax function]], or [[ReLU|rectifier function]]. The important characteristic of the activation function is that it provides a smooth transition as input values change, i.e. a small change in input produces a small change in output. The following refers to a collection of functions <math>\textstyle g_i</math> as a [[Vector (mathematics and physics)|vector]] <math>\textstyle g = (g_1, g_2, \ldots, g_n)</math>.

[[File:Ann_dependency_(graph).svg|link=https://en.wikipedia.org/wiki/File:Ann_dependency_(graph).svg|thumb|150x150px|ANN dependency graph]]

This figure depicts such a decomposition of <math>\textstyle f</math>, with dependencies between variables indicated by arrows. These can be interpreted in two ways.

Line 46:

The two views are largely equivalent. In either case, for this particular architecture, the components of individual layers are independent of each other (e.g., the components of <math>\textstyle g</math> are independent of each other given their input <math>\textstyle h</math>). This naturally enables a degree of parallelism in the implementation.

[[File:Recurrent_ann_dependency_graph.png|link=https://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.png|thumb|134x134px|Two separate depictions of the recurrent ANN dependency graph]]

Networks such as the previous one are commonly called [[Feedforward neural network|feedforward]], because their graph is a [[directed acyclic graph]]. Networks with [[Cycle (graph theory)|cycles]] are commonly called [[Recurrent neural network|recurrent]]. Such networks are commonly depicted in the manner shown at the top of the figure, where <math>\textstyle f</math> is shown as dependent upon itself. However, an implied temporal dependence is not shown.