Multilinear principal component analysis: Difference between revisions

Revision as of 01:27, 24 March 2020

Latest revision as of 05:40, 20 June 2025

(18 intermediate revisions by 12 users not shown)

Line 1:

{{short description|Multilinear extension of principal component analysis}}

{{context|date=June 2012}}

'''Multilinear principal component analysis''' ('''MPCA''') is a [[Multilinear algebra|multilinear]] extension of [[principal component analysis]] (PCA). MPCAthat is employedused into theanalyze analysis of nM-way arrays, i.e. a cube or hyper-cube of numbers, also informally referred to as a "data tensortensors". NM-way arrays may be decomposed, analyzed, or modeled by

* '''linear tensor models''', such as [[Tensor rank|CANDECOMP/Parafac]], or by

* '''multilinear tensor models''', such as '''multilinear principal component analysis (MPCA),'''<ref name=Vasilescu2002a/><ref name=Vasilescu2003/> or '''multilinear (tensor) independent component analysis (MICA), etc'''.<ref name=MPCA-MICA2005/>

The origin of MPCA can be traced back to the [[Tucker decomposition]]<ref>{{Cite journal|last1=Tucker| first1=Ledyard R

| authorlink1 = Ledyard R Tucker

| title = Some mathematical notes on three-mode factor analysis

| journal = [[Psychometrika]]

| volume = 31 | issue = 3 | pages = 279–311

|date=September 1966

| doi = 10.1007/BF02289464 | pmid = 5221127

}}</ref> and Peter Kroonenberg's "M-mode PCA/3-mode PCA" work.<ref name="Kroonenberg1980">P. M. Kroonenberg and J. de Leeuw, [https://doi.org/10.1007%2FBF02293599 Principal component analysis of three-mode data by means of alternating least squares algorithms], Psychometrika, 45 (1980), pp. 69–97.</ref> In 2000, De Lathauwer et al. restated Tucker and Kroonenberg's work in clear and concise numerical computational terms in their SIAM paper entitled "[[Multilinear Singular Value Decomposition]]",<ref name="DeLathauwer2000a">{{cite journal | last1 = Lathauwer | first1 = L.D. | last2 = Moor | first2 = B.D. | last3 = Vandewalle | first3 = J. | year = 2000 | title = A multilinear singular value decomposition | url = http://portal.acm.org/citation.cfm?id=354398 | journal = SIAM Journal on Matrix Analysis and Applications | volume = 21 | issue = 4| pages = 1253–1278 | doi = 10.1137/s0895479896305696 }}</ref> (HOSVD) and in their paper "On the Best Rank-1 and Rank-(R₁, R₂, ..., R_N ) Approximation of Higher-order Tensors".<ref name=DeLathauwer2000b>{{cite journal | last1 = Lathauwer | first1 = L. D. | last2 = Moor | first2 = B. D. | last3 = Vandewalle | first3 = J. | year = 2000 | title = On the best rank-1 and rank-(R1, R2, ..., RN ) approximation of higher-order tensors | url = http://portal.acm.org/citation.cfm?id=354405 | journal = SIAM Journal on Matrix Analysis and Applications | volume = 21 | issue = 4| pages = 1324–1342 | doi = 10.1137/s0895479898346995 }}</ref>

Historically, MPCA has been referred to as "M-mode PCA", a terminology which was coined by Peter Kroonenberg in 1980.<ref name="Kroonenberg1980"/> In 2005, [[Vasilescu]] and [[Demetri Terzopoulos|Terzopoulos]] introduced the '''Multilinear PCA'''<ref name="MPCA-MICA2005">M. A. O. Vasilescu, D. Terzopoulos (2005) [http://www.media.mit.edu/~maov/mica/mica05.pdf "Multilinear Independent Component Analysis"], "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, June 2005, vol.1, 547–553."</ref> terminology as a way to better differentiate between linear and multilinear tensordata decomposition,models asthat wellemployed as,2nd toorder better differentiate between the workstatistics<ref name="Vasilescu2002b"/><ref name="Vasilescu2002a"/><ref name="Vasilescu2003"/><ref name="Vasilescu2004"/> that computedversus 2ndhigher order statistics associatedto withcompute eacha data tensorset mode(axis),of andindependent subsequentcomponents workfor oneach Multilinearmode, Independentsuch Componentas Analysis'''Multilinear ICA'''<ref name="MPCA-MICA2005"/> that computed higher order statistics associated with each tensor mode/axis.

Circa 2001, Vasilescu reframed the data analysis, recognition and synthesis problems as multilinear tensor problems based on the insight that most observed data are the compositional consequence of several causal factors of data formation, and are well suited for multi-modal data tensor analysis. The power of the tensor framework was showcased by analyzing human motion joint angles, facial images or textures in terms of their causal factors of data formation in the following works: Human Motion Signatures<ref name="Vasilescu2002b">M.A.O. Vasilescu (2002) [http://www.media.mit.edu/~maov/motionsignatures/hms_icpr02_corrected.pdf "Human Motion Signatures: Analysis, Synthesis, Recognition," Proceedings of International Conference on Pattern Recognition (ICPR 2002), Vol. 3, Quebec City, Canada, Aug, 2002, 456–460.]</ref>

(CVPR 2001, ICPR 2002), face recognition – [[TensorFaces]],<ref name="Vasilescu2002a"/><ref name="Vasilescu2003"/>

(ECCV 2002, CVPR 2003, etc.) and computer graphics – [[TensorTextures]]<ref name="Vasilescu2004"/> (Siggraph 2004).

Multilinear PCA may be applied to compute the causal factors of data formation, or as signal processing tool on data tensors whose individual observation have either been vectorized,<ref name="Vasilescu2002b"/><ref name="Vasilescu2002a">M.A.O. Vasilescu, [[Demetri Terzopoulos|D. Terzopoulos]] (2002) [http://www.media.mit.edu/~maov/tensorfaces/eccv02_corrected.pdf "Multilinear Analysis of Image Ensembles: TensorFaces," Proc. 7th European Conference on Computer Vision (ECCV'02), Copenhagen, Denmark, May, 2002, in Computer Vision – ECCV 2002, Lecture Notes in Computer Science, Vol. 2350, A. Heyden et al. (Eds.), Springer-Verlag, Berlin, 2002, 447–460. ]</ref><ref name="Vasilescu2003">M.A.O. Vasilescu, D. Terzopoulos (2003) [http://www.media.mit.edu/~maov/tensorfaces/cvpr03.pdf "Multilinear Subspace Analysis for Image Ensembles,'' M. A. O. Vasilescu, D. Terzopoulos, Proc. Computer Vision and Pattern Recognition Conf. (CVPR '03), Vol.2, Madison, WI, June, 2003, 93–99.]</ref><ref name="Vasilescu2004">M.A.O. Vasilescu, D. Terzopoulos (2004) [http://www.media.mit.edu/~maov/tensortextures/Vasilescu_siggraph04.pdf "TensorTextures: Multilinear Image-Based Rendering", M. A. O. Vasilescu and D. Terzopoulos, Proc. ACM SIGGRAPH 2004 Conference Los Angeles, CA, August, 2004, in Computer Graphics Proceedings, Annual Conference Series, 2004, 336–342. ]</ref> or whose observations are treated as matrix<refa name="MPCA2008">{{citecollection journalof |column/row last1observations, =an Lu"observation |as first1a =matrix", H.and |concatenated last2into =a Plataniotis | first2 = K. N. | last3 = Venetsanopoulos | first3 = A. N. | year = 2008 | title = MPCA: Multilinear principal component analysis ofdata tensor objects | url = http://www.dsp.utoronto.ca/~haiping/Publication/MPCA_TNN08_rev2010.pdf | journalThe =latter IEEEapproach Trans.is Neuralsuitable Netw.for |compression volumeand =reducing 19redundancy |in issuethe =rows, 1|columns pagesand =fibers 18–39that |are doiunrelated =to 10.1109/tnn.2007.901277the |causal pmidfactors =of 18269936data | citeseerx = 10formation.1.1.331.5543 }}</ref> and concatenated into a data tensor.

Historically, MPCA has been referred to as "M-mode PCA", a terminology which was coined by Peter Kroonenberg in 1980.<ref name="Kroonenberg1980"/> In 2005, Vasilescu and [[Demetri Terzopoulos|Terzopoulos]] introduced the Multilinear PCA<ref name="MPCA-MICA2005">M. A. O. Vasilescu, D. Terzopoulos (2005) [http://www.media.mit.edu/~maov/mica/mica05.pdf "Multilinear Independent Component Analysis"], "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, June 2005, vol.1, 547–553."</ref> terminology as a way to better differentiate between linear and multilinear tensor decomposition, as well as, to better differentiate between the work<ref name="Vasilescu2002b"/><ref name="Vasilescu2002a"/><ref name="Vasilescu2003"/><ref name="Vasilescu2004"/> that computed 2nd order statistics associated with each data tensor mode(axis), and subsequent work on Multilinear Independent Component Analysis<ref name="MPCA-MICA2005"/> that computed higher order statistics associated with each tensor mode/axis.

Multilinear PCA may be applied to compute the causal factors of data formation, or as signal processing tool on data tensors whose individual observation have either been vectorized,<ref name="Vasilescu2002b"/><ref name="Vasilescu2002a">M.A.O. Vasilescu, [[Demetri Terzopoulos|D. Terzopoulos]] (2002) [http://www.media.mit.edu/~maov/tensorfaces/eccv02_corrected.pdf "Multilinear Analysis of Image Ensembles: TensorFaces," Proc. 7th European Conference on Computer Vision (ECCV'02), Copenhagen, Denmark, May, 2002, in Computer Vision – ECCV 2002, Lecture Notes in Computer Science, Vol. 2350, A. Heyden et al. (Eds.), Springer-Verlag, Berlin, 2002, 447–460. ]</ref><ref name="Vasilescu2003">M.A.O. Vasilescu, D. Terzopoulos (2003) [http://www.media.mit.edu/~maov/tensorfaces/cvpr03.pdf "Multilinear Subspace Analysis for Image Ensembles,'' M. A. O. Vasilescu, D. Terzopoulos, Proc. Computer Vision and Pattern Recognition Conf. (CVPR '03), Vol.2, Madison, WI, June, 2003, 93–99.]</ref><ref name="Vasilescu2004">M.A.O. Vasilescu, D. Terzopoulos (2004) [http://www.media.mit.edu/~maov/tensortextures/Vasilescu_siggraph04.pdf "TensorTextures: Multilinear Image-Based Rendering", M. A. O. Vasilescu and D. Terzopoulos, Proc. ACM SIGGRAPH 2004 Conference Los Angeles, CA, August, 2004, in Computer Graphics Proceedings, Annual Conference Series, 2004, 336–342. ]</ref> or whose observations are treated as matrix<ref name="MPCA2008">{{cite journal | last1 = Lu | first1 = H. | last2 = Plataniotis | first2 = K. N. | last3 = Venetsanopoulos | first3 = A. N. | year = 2008 | title = MPCA: Multilinear principal component analysis of tensor objects | url = http://www.dsp.utoronto.ca/~haiping/Publication/MPCA_TNN08_rev2010.pdf | journal = IEEE Trans. Neural Netw. | volume = 19 | issue = 1| pages = 18–39 | doi = 10.1109/tnn.2007.901277 | pmid = 18269936 | citeseerx = 10.1.1.331.5543 }}</ref> and concatenated into a data tensor.

}}</ref>Vasilescu and PeterTerzopoulos Kroonenberg'sin "M-modetheir PCA/3-modepaper PCA"[[TensorFaces]]"<ref work.name=Vasilescu2002a/><ref name="Kroonenberg1980Vasilescu2003"/>P. M.introduced Kroonenberg and J. de Leeuw,the [https://doi.org/10.1007%2FBF02293599[HOSVD| Principal component analysis of three'''M-mode dataSVD''']] byalgorithm meanswhich of alternating least squaresare algorithms], Psychometrika,misidentified 45in (1980),the pp.literature 69–97.</ref>as Inthe 2000, De Lathauwer et al. restated Tucker and Kroonenberg's work in clear and concise numerical computational terms in their SIAM paper entitled "[[Multilinear Singular Value Decomposition]]",''HOSVD'''<ref name="DeLathauwer2000a"DeLathauwer2000b>{{cite journal | last1 = Lathauwer | first1 = L. D. | last2 = Moor | first2 = B. D. | last3 = Vandewalle | first3 = J. | year = 2000 | title = AOn multilinearthe singularbest valuerank-1 decompositionand rank-(R1, R2, ..., RN ) approximation of higher-order tensors | url = http://portal.acm.org/citation.cfm?id=354398354405 | journal = SIAM Journal on Matrix Analysis and Applications | volume = 21 | issue = 4| pages = 1253–12781324–1342 | doi = 10.1137/s0895479896305696s0895479898346995 }}</ref>| (HOSVD) and in their paper "On the Best Rankurl-1access and= Rank-(R₁,subscription R<sub>2}}</subref>, ..., R_N ) Approximation of Higher-order Tensors".<ref name=DeLathauwer2000b"DeLathauwer2000a">{{cite journal | last1 = Lathauwer | first1 = L. D. | last2 = Moor | first2 = B. D. | last3 = Vandewalle | first3 = J. | year = 2000 | title = OnA themultilinear bestsingular rank-1value and rank-(R1, R2, ..., RN ) approximation of higher-order tensorsdecomposition | url = http://portal.acm.org/citation.cfm?id=354405354398 | journal = SIAM Journal on Matrix Analysis and Applications | volume = 21 | issue = 4| pages = 1324–13421253–1278 | doi = 10.1137/s0895479898346995s0895479896305696 | url-access = subscription }}</ref>

MPCA computes a set of orthonormal matrices associated with each mode of the data tensor which are analogous to the orthonormal row and column space of a matrix computed by the matrix SVD. This transformation aims to capture as high a variance as possible, accounting for as much of the variability in the data associated with each data tensor mode(axis).

or the '''Tucker''' which employ the power method or gradient descent, respectively.

CircaVasilescu 2001,and VasilescuTerzopoulos reframedframed the data analysis, recognition and synthesis problems as multilinear tensor problems. basedData onis theviewed insight that most observed data areas the compositional consequence of several causal factors of data formation, andthat are well suited for multi-modal datatensor tensorfactor analysis. The power of the tensor framework was showcased by analyzing human motion joint angles, facial images or textures in terms of their causal factors of data formation in the following workspapers: Human Motion Signatures<ref name="Vasilescu2002b">M.A.O. Vasilescu (2002) [http://www.media.mit.edu/~maov/motionsignatures/hms_icpr02_corrected.pdf "Human Motion Signatures: Analysis, Synthesis, Recognition," Proceedings of International Conference on Pattern Recognition (ICPR 2002), Vol. 3, Quebec City, Canada, Aug, 2002, 456–460.]</ref>

(CVPR 2001, ICPR 2002), face recognition – [[TensorFaces]],<ref name="Vasilescu2002a"/><ref name="Vasilescu2003"/>

(ECCV 2002, CVPR 2003, etc.) and computer graphics – [[TensorTextures]]<ref name="Vasilescu2004"/> (Siggraph 2004).

== The algorithm ==

The MPCA solution follows the alternating least square (ALS) approach.<ref name="Kroonenberg1980"/> It is iterative in nature. As in PCA, MPCA works on centered data. Centering is a little more complicated for tensors, and it is problem dependent.

As in PCA, MPCA works on centered data. Centering is a little more complicated for tensors, and it is problem dependent.

== Feature selection ==

MPCA features: Supervised MPCA feature selection is usedemployed in causal factor analysis that facilitates object recognition<ref name="MPCA">M. A. O. Vasilescu, D. Terzopoulos (2003) [http://www.cs.toronto.edu/~maov/tensorfaces/cvpr03.pdf "Multilinear Subspace Analysis of Image Ensembles"], "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’03), Madison, WI, June, 2003"</ref> while unsuperviseda semi-supervised MPCA feature selection is employed in visualization tasktasks.<ref>H. Lu, H.-L. Eng, M. Thida, and K.N. Plataniotis, "[http://www.dsp.utoronto.ca/~haiping/Publication/CrowdMPCA_CIKM2010.pdf Visualization and Clustering of Crowd Video Content in MPCA Subspace]," in Proceedings of the 19th ACM Conference on Information and Knowledge Management (CIKM 2010), Toronto, ON, Canada, October, 2010.</ref>

== Extensions ==

Various extensionsextension of MPCA have been developed:<ref>{{cite journal

|first=Haiping |last=Lu

|first2=K.N. |last2=Plataniotis

|first3=A.N. |last3=Venetsanopoulos

|url=http://www.dsp.utoronto.ca/~haiping/Publication/SurveyMSL_PR2011.pdf

|title=A Survey of Multilinear Subspace Learning for Tensor Data

|journal=Pattern Recognition

|volume=44 |number=7 |pages=1540–1551 |year=2011

|doi=10.1016/j.patcog.2011.01.004

}}</ref>

*Uncorrelated MPCA (UMPCA)<ref name="UMPCA">H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, "[http://www.dsp.utoronto.ca/~haiping/Publication/UMPCA_TNN09.pdf Uncorrelated multilinear principal component analysis for unsupervised multilinear subspace learning]," IEEE Trans. Neural Netw., vol. 20, no. 11, pp. 1820–1836, Nov. 2009.</ref> In contrast, the uncorrelated MPCA (UMPCA) generates uncorrelated multilinear features.<ref name="UMPCA"/>

*[[Boosting (meta-algorithm)|Boosting]]+MPCA<ref>H. Lu, K. N. Plataniotis and A. N. Venetsanopoulos, "[http://www.hindawi.com/journals/ivp/2009/713183.html Boosting Discriminant Learners for Gait Recognition using MPCA Features] {{webarchive|url=https://web.archive.org/web/20101022214324/http://www.hindawi.com/journals/ivp/2009/713183.html |date=2010-10-22 }}", EURASIP Journal on Image and Video Processing, Volume 2009, Article ID 713183, 11 pages, 2009. {{doi|10.1155/2009/713183}}.</ref>

*Non-negative MPCA (NMPCA)<ref>Y. Panagakis, C. Kotropoulos, G. R. Arce, "Non-negative multilinear principal component analysis of auditory temporal modulations for music genre classification", IEEE Trans. on Audio, Speech, and Language Processing, vol. 18, no. 3, pp. 576–588, 2010.</ref>

*Robust MPCA (RMPCA)<ref>K. Inoue, K. Hara, K. Urahama, "Robust multilinear principal component analysis", Proc. IEEE Conference on Computer Vision, 2009, pp. 591–597.</ref>

*Multi-Tensor Factorization, that also finds the number of components automatically (MTF)<ref>{{Cite journal|last=Khan|first=Suleiman A.|last2=Leppäaho|first2=Eemeli|last3=Kaski|first3=Samuel|date=2016-06-10|title=Bayesian multi-tensor factorization|journal=Machine Learning|language=en|volume=105|issue=2|pages=233–253|doi=10.1007/s10994-016-5563-y|issn=0885-6125|arxiv=1412.4679}}</ref>

Line 55 ⟶ 36:

[[Category:Dimension reduction]]

[[Category:Machine learning]]