Least-squares function approximation: Difference between revisions

Revision as of 08:50, 10 July 2013

Latest revision as of 01:46, 13 December 2023

(15 intermediate revisions by 13 users not shown)

Line 1:

{{Short description|Mathematical method}}

In [[mathematics]], the idea of '''least squares function approximation''' canapplies bethe appliedprinciple toof [[functionleast approximation|approximatingsquares]] a givento [[function approximation]], by means of a weighted sum of other functions. The best approximation can be defined as that which minimisesminimizes the difference between the original function and the approximation; for a least-squares approach the quality of the approximation is measured in terms of the squared differences between the two.

==Functional analysis==

{{See also|Fourier series|Generalized Fourier series}}

A generalization to approximation of a data set is the approximation of a function by a sum of other functions, usually an [[orthogonal functions|orthogonal set]]:<ref name=Lanczos>

{{cite book |title=Applied analysis |author=Cornelius Lanczos |pages =212–213 |isbn=0-486-65656-X |publisher=Dover Publications |year=1988 |edition=Reprint of 1956 Prentice–Hall |url=httphttps://books.google.com/books?id=6E85hExIqHYC&pg=PA212}}

</ref>

:<math>f(x) \approx f_n (x) = a_1 \phi _1 (x) + a_2 \phi _2(x) + \cdots + a_n \phi _n (x), \ </math>

with the set of functions {<math>\ \phi _j (x) </math>} an [[Orthonormal_set#Real-valued_functions|orthonormal set]] over the interval of interest, {{nowrap|say [a, b]}}: see also [[Fejér's theorem]]. The coefficients {<math>\ a_j </math>} are selected to make the magnitude of the difference ||{{nowrap|''f'' − ''f''_''n'' }}||² as small as possible. For example, the magnitude, or norm, of a function {{nowrap|''g'' (''x'' )}} over the {{nowrap|interval [a, b]}} can be defined by:<ref name=Folland>

{{cite book |title=Fourier analysis and its application |page =69 |chapter=Equation 3.14 |author=Gerald B Folland |chapter-url=httphttps://books.google.com/books?id=ix2iCQ-o9x4C&pg=PA69 |isbn=978-0-8218-4790-29 |publisher=American Mathematical Society Bookstore |year=2009 |edition=Reprint of Wadsworth and Brooks/Cole 1992}}

</ref>

Line 20:

:<math> \|g\| = \left(\int_a^b g^*(x)g(x) \, dx \right)^{1/2} </math>

where the ‘*’ denotes [[complex conjugate]] in the case of complex functions. The extension of Pythagoras' theorem in this manner leads to [[function space]]s and the notion of [[Lebesgue measure]], an idea of “space” more general than the original basis of Euclidean geometry. The {{nowrap|{ <math>\phi_j (x)\ </math> } }} satisfy [[Orthogonal#Orthogonal_functions|orthonormality relations]]:<ref name=Folland2>

{{cite book |title=Fourier Analysis and Its Applications|page =69 |first1=Gerald B | last1= Folland|url=httphttps://books.google.com/books?id=ix2iCQ-o9x4C&pg=PA69 |isbn=978-0-8218-4790-29 |year=2009 |publisher=American Mathematical Society }}

</ref>

:<math> \int_a^b \phi _i^* (x)\phi _j (x) \, dx =\delta_{ij},</math>

where ''δ''_''ij'''' is the [[Kronecker delta]]. Substituting function {{nowrap|''f''_''n''}} into these equations then leads to

the ''n''-dimensional [[Pythagorean theorem]]:<ref name=Wood>

{{cite book |title=Statistical methods: the geometric approach |author= David J. Saville, Graham R. Wood |chapter=§2.5 Sum of squares |page=30 |chapter-url=httphttps://books.google.com/books?id=8ummgMVRev0C&pg=PA30 |isbn=0-387-97517-9 |year=1991 |edition=3rd |publisher=Springer}}

</ref>

:<math>\|f_n\|^2 = |a_1|^2 + |a_2|^2 + \cdots + |a_n|^2. \, </math>

The coefficients {''a''_''j''''} making {{nowrap begin}}||''f'' − ''f''_''n''||²{{nowrap end}} as small as possible are found to be:<ref name=Lanczos/>

:<math>a_j = \int_a^b \phi _j^* (x)f (x) \, dx. </math>

Line 40:

The generalization of the ''n''-dimensional Pythagorean theorem to ''infinite-dimensional&thinsp;'' [[real number|real]] inner product spaces is known as [[Parseval's identity]] or Parseval's equation.<ref name=Folland3>

{{cite book |title=cited work |page =77 |chapter=Equation 3.22 |author=Gerald B Folland |chapter-url=httphttps://books.google.com/books?id=ix2iCQ-o9x4C&pg=PA77 |isbn=978-0-8218-4790-29 |date=2009-01-13 |publisher =American Mathematical Soc. }}

</ref> Particular examples of such a representation of a function are the [[Fourier series]] and the [[generalized Fourier series]].

==Further discussion==

===Using linear algebra===

It follows that one can find a "best" approximation of another function by minimizing the area between two functions, a continuous function <math>f</math> on <math>[a,b]</math> and a function <math>g\in W</math> where <math>W</math> is a subspace of <math>C[a,b]</math>:

:<math>\text{Area} = \int_a^b \left\vert f(x) - g(x)\right\vert \, dx,</math>

all within the subspace <math>W</math>. Due to the frequent difficulty of evaluating integrands involving absolute value, one can instead define

:<math>\int_a^b [ f(x) - g(x) ] ^2\, dx</math>

as an adequate criterion for obtaining the least squares approximation, function <math>g</math>, of <math>f</math> with respect to the inner product space <math>W</math>.

As such, <math>\lVert f-g \rVert ^2</math> or, equivalently, <math>\lVert f-g \rVert</math>, can thus be written in vector form:

:<math>\int_a^b [ f(x)-g(x) ]^2\, dx = \left\langle f-g , f-g\right\rangle = \lVert f-g\rVert^2.</math>

In other words, the least squares approximation of <math>f</math> is the function <math>g\in \text{ subspace } W</math> closest to <math>f</math> in terms of the inner product <math>\left \langle f,g \right \rangle</math>. Furthermore, this can be applied with a theorem:

:Let <math>f</math> be continuous on <math>[ a,b ]</math>, and let <math>W</math> be a finite-dimensional subspace of <math>C[a,b]</math>. The least squares approximating function of <math>f</math> with respect to <math>W</math> is given by

::<math>g = \left \langle f,\vec w_1 \right \rangle \vec w_1 + \left \langle f,\vec w_2 \right \rangle \vec w_2 + \cdots + \left \langle f,\vec w_n \right \rangle \vec w_n,</math>

:where <math>B = \{\vec w_1 , \vec w_2 , \dots , \vec w_n \}</math> is an orthonormal basis for <math>W</math>.

==References==