Levenberg–Marquardt algorithm: Difference between revisions

Revision as of 22:10, 2 January 2022

Revision as of 14:05, 24 January 2022

Line 1:

{{short description|Algorithm used to solve non-linear least squares problems}}

In [[mathematics]] and computing, the '''Levenberg–Marquardt algorithm''' ('''LMA''' or just '''LM'''), also known as the '''damped least-squares''' ('''DLS''') method, is used to solve [[non-linear least squares]] problems. These minimization problems arise especially in [[least squares]] [[curve fitting]]. Applied toThe [[neuralLMA network|artificialinterpolates neuralbetween networkthe [[Gauss–Newton algorithm]] training,(GNA) aand Levenberg-Marquardtthe algorithmmethod oftenof converges[[gradient fasterdescent]]. thanThe first-orderLMA is more [[backpropagationRobustness (computer science)|robust]] methods.<ref>{{citethan journal|title=Improvedthe ComputationGNA, forwhich Levenberg–Marquardtmeans Training|last1=Wiliamowski|first1=Bogdan|last2=Yu|first2=Hao|journal=IEEEthat Transactionsin onmany Neuralcases Networksit finds a solution even if it starts very far off the final minimum. For well-behaved functions and Learningreasonable Systems|volume=21|issue=6|date=Junestarting 2010|url=https://wwwparameters, the LMA tends to be slower than the GNA.eng.auburn.edu/~wilambm/pap/2010/Improved%20Computation%20for%20LM%20Training LMA can also be viewed as [[Gauss–Newton]] using a [[trust region]] approach.pdf}}</ref>

The LMA is used in many software applications for solving generic curve-fitting problems. However, as with many fitting algorithms, the LMA finds only a [[local minimum]], which is not necessarily the [[global minimum]]. The LMA interpolates between the [[Gauss–Newton algorithm]] (GNA) and the method of [[gradient descent]]. The LMA is more [[Robustness (computer science)|robust]] than the GNA, which means that in many cases it finds a solution even if it starts very far off the final minimum. For well-behaved functions and reasonable starting parameters, the LMA tends to be slower than the GNA. LMA can also be viewed as [[Gauss–Newton]] using a [[trust region]] approach.

The algorithm was first published in 1944 by [[Kenneth Levenberg]],<ref name="Levenberg"/> while working at the [[Frankford Arsenal|Frankford Army Arsenal]]. It was rediscovered in 1963 by [[Donald Marquardt]],<ref name="Marquardt"/> who worked as a [[statistician]] at [[DuPont]], and independently by Girard,<ref name="Girard"/> Wynne<ref name="Wynne"/> and Morrison.<ref name="Morrison"/>

The LMA is used in many software applications for solving generic curve-fitting problems. By using the Gauss-Newton algorithm it often converges faster than first-order methods.<ref>{{cite journal|title=Improved Computation for Levenberg–Marquardt Training|last1=Wiliamowski|first1=Bogdan|last2=Yu|first2=Hao|journal=IEEE Transactions on Neural Networks and Learning Systems|volume=21|issue=6|date=June 2010|url=https://www.eng.auburn.edu/~wilambm/pap/2010/Improved%20Computation%20for%20LM%20Training.pdf}}</ref> However, like other iterative optimization algorithms, the LMA finds only a [[local minimum]], which is not necessarily the [[global minimum]].

== The problem ==