Functional regression

This sandbox is in the article namespace. Either move this page into your userspace, or remove the {{User sandbox}} template. Functional regression is a version of the regression analysis when responses or covariates include functional data. One the one hand, functional regression models can be classified into four types depending on whether the response or covariates are functional or scalar: (i) scalar response with functional covariates, (ii) functional response with scalar covariates, (iii) functional response with functional covariates, and (iv) scalar or functional response with functional and scalar covariates. On the other hand, functional regression models can be linear, partially linear, or nonlinear. In particular, functional polynomial models, functional single and multiple single models and functional additive models are three special cases of functional nonlinear models.

Functional linear models (FLMs)

Functional linear models (FLMs) are an extension of linear regression with scalar response ${\displaystyle Y\in \mathbb {R} }$  and scalar covariates ${\displaystyle X\in \mathbb {R} ^{p}}$ , which can be written as $${\displaystyle Y=\beta _{0}+\langle X,\beta \rangle +\epsilon ,}$$  where ${\displaystyle \langle \cdot ,\cdot \rangle }$  denotes the inner product in Euclidean space, ${\displaystyle \beta _{0}\in \mathbb {R} }$  and ${\displaystyle \beta \in \mathbb {R} ^{p}}$  denote the regression coefficients, and ${\displaystyle \epsilon }$  is a random error with mean zero and finite variance. FLMs can be divided into three types based on responses and covariates.

Functional linear models with scalar response

Functional linear models with scalar response (also known as functional linear regression (FLR)) can are obtained by replacing the scalar covariates ${\displaystyle X}$  and the coefficient vector ${\displaystyle \beta }$  in the traditional multivariate linear model by a centered functional covariate ${\displaystyle X^{c}(t)=X(t)-\mathbb {E} (X(t))}$  and a coefficient function ${\displaystyle \beta =\beta (t)}$  for ${\displaystyle t\in {\mathcal {T}}}$ , respectively,

$${\displaystyle Y=\beta _{0}+\langle X^{c},\beta \rangle +\epsilon =\beta _{0}+\int _{\mathcal {T}}X^{c}(t)\beta (t)dt+\epsilon ,}$$

1

where ${\displaystyle \langle \cdot ,\cdot \rangle }$  here denotes the inner product in ${\displaystyle L^{2}}$  space. One approach to estimating ${\displaystyle \beta _{0}}$  and ${\displaystyle \beta (t)}$  is to expand the covariate ${\displaystyle X}$  and the coefficient function ${\displaystyle \beta (t)}$  in the same functional basis, such as B-spline basis or the eigenfunctions in the Karhunen–Loève expansion. Suppose ${\displaystyle \{\phi _{k}\}_{k=1}^{\infty }}$  is an orthonormal basis of ${\displaystyle L^{2}}$  space. Expanding ${\displaystyle X}$  and ${\displaystyle \beta }$  in this basis, ${\displaystyle X^{c}(t)=\sum _{k=1}^{\infty }x_{k}\phi _{k}(t)}$ , ${\displaystyle \beta (t)=\sum _{k=1}^{\infty }\beta _{k}\phi _{k}(t)}$ , model (1) becomes $${\displaystyle Y=\beta _{0}+\sum _{k=1}^{\infty }\beta _{k}x_{k}+\epsilon ,}$$  where in implementation the infinite sum is replaced by a finite sum truncated at ${\displaystyle K}$  $${\displaystyle Y=\beta _{0}+\sum _{k=1}^{K}\beta _{k}x_{k}+\epsilon }$$  where ${\displaystyle K\in \mathbb {N} }$  is finite^[1].
Adding multiple functional and scalar covariates, the FLR can be extended as $${\displaystyle Y=\langle \mathbf {Z} ,\alpha \rangle +\sum _{j=1}^{p}\int _{{\mathcal {T}}_{j}}X_{j}^{c}(t)\beta _{j}(t)dt+\epsilon }$$  where ${\displaystyle \mathbf {Z} =(Z_{1},\cdots ,Z_{q})^{T}}$  with ${\displaystyle Z_{1}=1}$  is a vector of scalar covariates, ${\displaystyle \alpha =(\alpha _{1},\cdots ,\alpha _{q})^{T}}$  is a vector of coefficients corresponding to ${\displaystyle \mathbf {Z} }$ , ${\displaystyle \langle \cdot ,\cdot \rangle }$  denotes the inner product in Euclidean space, ${\displaystyle X_{1}^{c},\cdots ,X_{p}^{c}}$  are multiple centered functional covariates given by ${\displaystyle X_{j}^{c}(\cdot )=X_{j}(\cdot )-\mathbb {E} (X_{j}(\cdot ))}$ , and ${\displaystyle {\mathcal {T}}_{j}}$  is the interval ${\displaystyle X_{j}(\cdot )}$  is defined on. However, due to the parametric component ${\displaystyle \alpha }$ , the estimation of this model is different from that of the FLR. A possible approach to estimating ${\displaystyle \alpha }$  is through generalized estimating equation with the nonparametric part ${\displaystyle \sum _{j=1}^{p}\int _{{\mathcal {T}}_{j}}X_{j}^{c}(t)\beta _{j}(t)dt}$  replaced by its estimate for a given ${\displaystyle \alpha }$ ^[2]. Once ${\displaystyle \alpha }$  is estimated, one can apply any suitable consistent method to ${\displaystyle Y-\langle \mathbf {Z} ,{\hat {\alpha }}\rangle }$  to estimate ${\displaystyle \beta _{j}}$ s^[1].

Functional linear models with functional response

For a function ${\displaystyle Y(\cdot )}$  on ${\displaystyle {\mathcal {T}}_{Y}}$  and a functional covariate ${\displaystyle X(\cdot )}$  on ${\displaystyle {\mathcal {T}}_{X}}$ , two primary models have been considered^[1][3]. One functional linear model regressing ${\displaystyle Y(\cdot )}$  on ${\displaystyle X(\cdot )}$  is given by $${\displaystyle Y(s)=\beta _{0}(s)+\int _{{\mathcal {T}}_{X}}\beta (s,t)X^{c}(t)dt+\epsilon (s)}$$  where ${\displaystyle s\in {\mathcal {T}}_{Y}}$ , ${\displaystyle t\in {\mathcal {T}}_{X}}$ , ${\displaystyle X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))}$  is still the centered functional covariate, ${\displaystyle \beta _{0}(\cdot )}$  and ${\displaystyle \beta (\cdot ,\cdot )}$  are coefficient functions, and ${\displaystyle \epsilon (\cdot )}$  is usually assumed to be a Gaussian process with mean zero. In this case, at any given time ${\displaystyle s\in {\mathcal {T}}_{Y}}$ , the value of ${\displaystyle Y}$ , i.e. ${\displaystyle Y(s)}$ , depends on the entire trajectory of ${\displaystyle X}$ . This model, for any given time ${\displaystyle s}$ , is an extension of the traditional multivariate linear regression model by simply replacing the inner product in Euclidean space by that in ${\displaystyle L^{2}}$  space. Thus, estimation of this model can be given by analogy to multivariate linear regression $${\displaystyle r_{XY}=R_{XX}\beta ,{\text{ for }}\beta \in L^{2}({\mathcal {T}}_{X}\times {\mathcal {T}}_{X})}$$  where ${\displaystyle r_{XY}(s,t)={\text{cov}}(X(s),Y(t))}$ , ${\displaystyle R_{XX}:L^{2}\times L^{2}\rightarrow L^{2}\times L^{2}}$  is defined as ${\displaystyle (R_{XX}\beta )(s,t)=\int r_{XX}(s,w)\beta (w,t)dw}$  with ${\displaystyle r_{XX}(s,t)={\text{cov}}(X(s),X(t))}$ . Furthermore, regularization is needed because ${\displaystyle R_{XX}}$  is a compact operator and its inverse is not bounded^[1].
In particular, taking ${\displaystyle X(\cdot )}$  as a constant function gives a special case of this model $${\displaystyle Y(s)=\sum _{j=1}^{p}X_{j}\beta _{j}(s)+\epsilon (s)}$$  which is a FLM with functional response and scalar covariates.

Concurrent models

Assuming that ${\displaystyle {\mathcal {T}}_{X}={\mathcal {T}}_{Y}:={\mathcal {T}}}$ , another model called varying-coefficient model is of the form $${\displaystyle Y(s)=\alpha _{0}(s)+\alpha (s)X(s)+\epsilon (s)}$$  Note that this model assumes the value of ${\displaystyle Y}$  at time ${\displaystyle s}$ , i.e. ${\displaystyle Y(s)}$ , only depends on that of ${\displaystyle X}$  at the same time, ${\displaystyle X(s)}$ , and thus is a concurrent regression model. A possible way to estimate ${\displaystyle \alpha }$  is a two-step procedure: (i) For any ${\displaystyle s\in {\mathcal {T}}}$  fixed, an estimate of ${\displaystyle \alpha (s)}$  can be computed by applying ordinary least squares to a neighborhood of ${\displaystyle s}$ . Let the corresponding estimate be denoted by ${\displaystyle {\tilde {\alpha }}(s)}$ . (ii) The final estimate ${\displaystyle {\hat {\alpha }}}$  is then obtained by smoothing ${\displaystyle {\tilde {\alpha }}(s)}$  with respect to ${\displaystyle s}$ ^[1].

Functional nonlinear models

Functional polynomial models

Functional polynomial models is an extension of the FLMs, analogous to extending multivariate linear models to polynomial ones. For a scalar response ${\displaystyle Y}$  and a functional covariate ${\displaystyle X(\cdot )}$  defined on an interval ${\displaystyle {\mathcal {T}}}$ , the simplest example of functional polynomial models is functional quadratic regression^[4] $${\displaystyle Y=\alpha +\int _{\mathcal {T}}\beta (t)X^{c}(t)dt+\int _{\mathcal {T}}\int _{\mathcal {T}}\gamma (s,t)X^{c}(s)X^{c}(t)dsdt+\epsilon }$$  where ${\displaystyle X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))}$  is the centered functional covariate, ${\displaystyle \alpha }$  is a scalar coefficient, ${\displaystyle \beta (\cdot )}$  and ${\displaystyle \gamma (\cdot ,\cdot )}$  are coefficient functions defined on ${\displaystyle {\mathcal {T}}}$  and ${\displaystyle {\mathcal {T}}\times {\mathcal {T}}}$  respectively, and ${\displaystyle \epsilon }$  is a random error with mean zero and variance finite. By analogy to FLMs, estimation of functional polynomial models can be obtained through expanding both the centered covariate ${\displaystyle X^{c}}$  and the coefficient functions ${\displaystyle \beta }$  and ${\displaystyle \gamma }$  on an orthonormal basis. Then the model can be equivalently written as multivariate polynomial regression and thus the corresponding estimation is straightforward.

Functional single and multiple index models

A functional multiple index model is given by $${\displaystyle Y=g\left(\int _{\mathcal {T}}X^{c}(t)\beta _{1}(t)dt,\cdots ,\int _{\mathcal {T}}X^{c}(t)\beta _{p}(t)dt\right)+\epsilon .}$$  Taking ${\displaystyle p=1}$  yields a functional single index model. However, this model is problematic due to curse of dimensionality. In other words, with ${\displaystyle p>1}$  and relatively small sample sizes, this model often leads to high variability of the estimator^[5]. Alternatively, a preferable ${\displaystyle p}$ -component functional multiple index model can be formed as $${\displaystyle Y=g_{1}\left(\int _{\mathcal {T}}X^{c}(t)\beta _{1}(t)dt\right)+\cdots +g_{p}\left(\int _{\mathcal {T}}X^{c}(t)\beta _{p}(t)dt\right)+\epsilon .}$$ 

Functional additive models

Given an expansion of a functional covariate ${\displaystyle X}$  on an orthonormal basis ${\displaystyle \{\phi _{k}\}_{k=1}^{\infty }}$ : ${\displaystyle X(t)=\sum _{k=1}^{\infty }x_{k}\phi _{k}(t)}$ , a functional linear model with scalar response as stated before can be written as $${\displaystyle \mathbb {E} (Y|X)=\mathbb {E} (Y)+\sum _{k=1}^{\infty }\beta _{k}x_{k}.}$$  A functional additive model can be given by replacing the linear function of ${\displaystyle x_{k}}$  by a general smooth function ${\displaystyle f_{k}}$  $${\displaystyle \mathbb {E} (Y|X)=\mathbb {E} (Y)+\sum _{k=1}^{\infty }f_{k}(x_{k})}$$  where ${\displaystyle f_{k}}$  satisfies ${\displaystyle \mathbb {E} (f_{k}(x_{k}))=0}$  for ${\displaystyle k\in \mathbb {N} }$ ^[1].

Extensions

A direct extension of functional linear models with scalar response is to add a link function to create a generalized functional linear model (GFLM) by analogy to extending linear regression to generalized linear regression $${\displaystyle Y=g\left(\beta _{0}+\int _{\mathcal {T}}X^{c}(t)\beta (t)dt\right)+\epsilon }$$  where ${\displaystyle g}$  is a pre-specific link function.

See also

• Functional data analysis
• Functional principal component analysis
• Karhunen–Loève theorem
• Generalized linear model
• Generalized functional linear model
• Stochastic processes
• Lp space

References

1. Wang, Chiou and Müller (2016). "Functional data analysis". Annual Review of Statistics and Its Application. 3:257–295. doi:10.1146/annurev-statistics-041715-033624
2. Hu, Wang and Carroll (2004). "Profile-kernel versus backfitting in the partially linear models for longitudinal/clustered data". Biometrika. 91 (2): 251–262. doi:10.1093/biomet/91.2.251
3. Ramsay and Silverman (2005). Functional data analysis, 2nd ed., New York : Springer, ISBN 0-387-40080-X
4. Yao and Müller (2010). "Functional quadratic regression". Biometrika. 97 (1):49–64. http://www.jstor.org/stable/27798896
5. Chen, Hall and Müller (2011). "Single and multiple index functional regression models with nonparametric link". The Annals of Statistics. 39 (3):1720–1747. http://www.jstor.org/stable/23033613

Further reading

• Morris (2015). Functional regression. Annual Review of Statistics and Its Application. 2:321–359. doi:10.1146/annurev-statistics-010814-020413.