Automatic differentiation: Difference between revisions

Revision as of 18:42, 26 March 2016

Revision as of 18:48, 26 March 2016

Line 140:

Reverse accumulation is more efficient than forward accumulation for functions {{math|''f'' : ℝ^''n'' → ℝ^''m''}} with {{math|''m'' ≪ ''n''}} as only {{math|''m''}} sweeps are necessary, compared to {{math|''n''}} sweeps for forward accumulation.

Reverse mode AD was first published in 1970 by [[Seppo Linnainmaa]] in his master thesis.<ref name="lin1970">Linnainmaa, S.Seppo (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 6-7.</ref><ref name="lin1976">Linnainmaa, S.Seppo (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.</ref><ref name="grie2012">Griewank, A.Andreas (2012). Who Invented the Reverse Mode of Differentiation?. Optimization Stories, Documenta Matematica, Extra Volume ISMP (2012), 389-400.</ref>

[[Backpropagation]] of errors in multilayer perceptrons, a technique used in [[machine learning]], is a special case of reverse mode AD.