Non-negative matrix factorization

Non-negative matrix factorization (NMF) is a group of algorithms in multivariate analysis and linear algebra where a matrix, ${\displaystyle \mathbf {X} }$, is factorized into (usually) two matrices, ${\displaystyle \mathbf {W} }$ and ${\displaystyle \mathbf {H} }$

${\displaystyle \operatorname {nmf} (\mathbf {X} )\rightarrow \mathbf {WH} }$

Factorization of matrices is generally non-unique, and a number of different methods of doing so have been developed (e.g. principal component analysis and singular value decomposition) by incorporating different constraints; non-negative matrix factorization differs from these methods in that it enforces the constraint that all three matrices must be non-negative, i.e., all elements must be equal to or greater than zero.

Usually the number of columns of W and the number of rows of H in NMF are selected so the product WH will become an approximation to X (it has been suggested that the NMF model should be called nonnegative matrix approximation instead). The full decomposition of X then amounts to the two non-negative matrices W and H as well as a residual U:

${\displaystyle \mathbf {X} =\mathbf {WH+U} }$

The elements of the residual matrix are can either be negative and positive - at least in the typical application of NMF.

Early work research on non-negative matrix factorizations was performed by a Finnish group of researchers in the middle of the 1990s under the name positive matrix factorization. It became more widely known after Lee and Seung's investigations of the properties of the algorithm, and after they published a simple useful algorithm.

There are different types of non-negative matrix factorizations and one of these is related to probabilistic latent semantic analysis and the latent class model. They different types arise from using different cost functions (divergence functions) and/or by regularization of the W and/or H matrices^[1].

Uniqueness

The factorization is not unique: A matrix and its inverse can be used to transform the two factorization matrices by, e.g.,

${\displaystyle \mathbf {WH} =\mathbf {WBB} ^{-1}\mathbf {H} }$ 

If the two new matrices ${\displaystyle \mathbf {{\tilde {W}}=WB} }$  and ${\displaystyle \mathbf {\tilde {H}} =\mathbf {B} ^{-1}\mathbf {H} }$  are non-negative they form another parametrization of the factorization.

The non-negativity of ${\displaystyle \mathbf {\tilde {W}} }$  and ${\displaystyle \mathbf {\tilde {H}} }$  applies at least if B is a non-negative monomial matrix. In this simple case it will just correspond to a scaling and a permutation.

More control over the non-uniqueness of NMF is obtained with sparsity contraints^[2].

Sources and external links

• J. Shen, G. W. Israël, "A receptor model using a specific non-negative transformation technique for ambient aerosol", Atmospheric Environment, 23(10):2289-2298, 1989.
• P. Paatero, U. Tapper, "Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values", Environmetrics, 5:111-126, 1994.
• Pia Anttila, Pentti Paatero, Unto Tapper, Olli Järvinen. "Source identification of bulk wet deposition in Finland by positive matrix factorization", Atmospheric Environment, 29(14):1705-1718, 1995
• Pentti Paatero, "Least squares formulation of robust non-negative factor analysis", Chemometrics and Intelligent Laboratory Systems, 37(1):23-35, 1997 May.
• Daniel D. Lee and H. Sebastian Seung, "Learning the parts of objects by non-negative matrix factorization", Nature, 401(6755):788-791, 1999 October.
• Daniel D. Lee and H. Sebastian Seung, "Algorithms for Non-negative Matrix Factorization", Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, 556-562, MIT Press, 2001.

References

1. Inderjit S. Dhillon, Suvrit Sra, "Generalized Nonnegative Matrix Approximations with Bregman Divergences", NIPS, 2005.
2. Julian Eggert, Edgar Körner, "Sparse coding and NMF", Proceedings. 2004 IEEE International Joint Conference on Neural Networks, 2004., pp. 2529-2533, 2004.