Content deleted Content added
Rescuing 1 sources and tagging 0 as dead.) #IABot (v2.0.9.5 |
|||
| (39 intermediate revisions by 15 users not shown) | |||
Line 1:
{{Short description|Class of nonparametric methods}}
In [[machine learning]], the '''kernel embedding of distributions''' (also called the '''kernel mean''' or '''mean map''') comprises a class of [[nonparametric]] methods in which a [[probability distribution]] is represented as an element of a [[reproducing kernel Hilbert space]] (RKHS).<ref name = "Smola2007">A. Smola, A. Gretton, L. Song, B. Schölkopf. (2007). [http://eprints.pascal-network.org/archive/00003987/01/SmoGreSonSch07.pdf A Hilbert Space Embedding for Distributions]. ''Algorithmic Learning Theory: 18th International Conference''. Springer: 13–31.</ref> A generalization of the individual data-point feature mapping done in classical [[kernel methods]], the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space operations such as [[inner product]]s, distances, [[projection (linear algebra)|projections]], [[linear transformation]]s, and [[spectral theory|spectral analysis]].<ref name = "Song2013">L. Song, K. Fukumizu, F. Dinuzzo, A. Gretton (2013). [http://www.gatsby.ucl.ac.uk/~gretton/papers/SonFukGre13.pdf Kernel Embeddings of Conditional Distributions: A unified kernel framework for nonparametric inference in graphical models]. ''IEEE Signal Processing Magazine'' '''30''': 98–111.</ref> This [[machine learning|learning]] framework is very general and can be applied to distributions over any space <math>\Omega </math> on which a sensible [[kernel function]] (measuring similarity between elements of <math>\Omega </math>) may be defined. For example, various kernels have been proposed for learning from data which are: [[Vector (mathematics and physics)|vectors]] in <math>\mathbb{R}^d</math>, discrete classes/categories, [[string (computer science)|string]]s, [[Graph (discrete mathematics)|graph]]s/[[network theory|networks]], images, [[time series]], [[manifold]]s, [[dynamical systems]], and other structured objects.<ref>J. Shawe-Taylor, N. Christianini. (2004). ''Kernel Methods for Pattern Analysis''. Cambridge University Press, Cambridge, UK.</ref><ref>T. Hofmann, B. Schölkopf, A. Smola. (2008). [http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1211819561 Kernel Methods in Machine Learning]. ''The Annals of Statistics'' '''36'''(3):1171–1220.</ref> The theory behind kernel embeddings of distributions has been primarily developed by [http://alex.smola.org/ Alex Smola], [http://www.cc.gatech.edu/~lsong/ Le Song ], [http://www.gatsby.ucl.ac.uk/~gretton/ Arthur Gretton], and [[Bernhard Schölkopf]]. A review of recent works on kernel embedding of distributions can be found in.<ref>{{Cite journal|last=Muandet|first=Krikamol|last2=Fukumizu|first2=Kenji|last3=Sriperumbudur|first3=Bharath|last4=Schölkopf|first4=Bernhard|date=2017-06-28|title=Kernel Mean Embedding of Distributions: A Review and Beyond|journal=Foundations and Trends in Machine Learning|language=English|volume=10|issue=1–2|pages=1–141|doi=10.1561/2200000060|issn=1935-8237|arxiv=1605.09522}}</ref>▼
{{cleanup|reason=This nonsense of calling a distribution ''P''(''X''), with a capital ''X'', when capital ''X'' is also the name of the random variable, and other like things, need to get cleaned up.|date=March 2020}}
▲In [[machine learning]], the '''kernel embedding of distributions''' (also called the '''kernel mean''' or '''mean map''') comprises a class of [[nonparametric]] methods in which a [[probability distribution]] is represented as an element of a [[reproducing kernel Hilbert space]] (RKHS).<ref name = "Smola2007">A. Smola, A. Gretton, L. Song, B. Schölkopf. (2007). [http://eprints.pascal-network.org/archive/00003987/01/SmoGreSonSch07.pdf A Hilbert Space Embedding for Distributions] {{Webarchive|url=https://web.archive.org/web/20131215111545/http://eprints.pascal-network.org/archive/00003987/01/SmoGreSonSch07.pdf |date=2013-12-15 }}. ''Algorithmic Learning Theory: 18th International Conference''. Springer: 13–31.</ref> A generalization of the individual data-point feature mapping done in classical [[kernel methods]], the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space operations such as [[inner product]]s, distances, [[projection (linear algebra)|projections]], [[linear transformation]]s, and [[spectral theory|spectral analysis]].<ref name = "Song2013">L. Song, K. Fukumizu, F. Dinuzzo, A. Gretton (2013). [http://www.gatsby.ucl.ac.uk/~gretton/papers/SonFukGre13.pdf Kernel Embeddings of Conditional Distributions: A unified kernel framework for nonparametric inference in graphical models]. ''IEEE Signal Processing Magazine'' '''30''': 98–111.</ref> This [[machine learning|learning]] framework is very general and can be applied to distributions over any space <math>\Omega </math> on which a sensible [[kernel function]] (measuring similarity between elements of <math>\Omega </math>) may be defined. For example, various kernels have been proposed for learning from data which are: [[Vector (mathematics and physics)|vectors]] in <math>\mathbb{R}^d</math>, discrete classes/categories, [[string (computer science)|string]]s, [[Graph (discrete mathematics)|graph]]s/[[network theory|networks]], images, [[time series]], [[manifold]]s, [[dynamical systems]], and other structured objects.<ref>J. Shawe-Taylor, N. Christianini. (2004). ''Kernel Methods for Pattern Analysis''. Cambridge University Press, Cambridge, UK.</ref><ref>T. Hofmann, B. Schölkopf, A. Smola. (2008). [http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1211819561 Kernel Methods in Machine Learning]. ''The Annals of Statistics'' '''36'''(3):1171–1220.</ref> The theory behind kernel embeddings of distributions has been primarily developed by [http://alex.smola.org/ Alex Smola], [http://www.cc.gatech.edu/~lsong/ Le Song ] {{Webarchive|url=https://web.archive.org/web/20210412002513/https://www.cc.gatech.edu/~lsong/ |date=2021-04-12 }}, [http://www.gatsby.ucl.ac.uk/~gretton/ Arthur Gretton], and [[Bernhard Schölkopf]]. A review of recent works on kernel embedding of distributions can be found in.<ref>{{Cite journal|last=Muandet|first=Krikamol|last2=Fukumizu|first2=Kenji|last3=Sriperumbudur|first3=Bharath|last4=Schölkopf|first4=Bernhard|date=2017-06-28|title=Kernel Mean Embedding of Distributions: A Review and Beyond|journal=Foundations and Trends in Machine Learning|language=English|volume=10|issue=1–2|pages=1–141|doi=10.1561/2200000060|issn=1935-8237|arxiv=1605.09522}}</ref>
The analysis of distributions is fundamental in [[machine learning]] and [[statistics]], and many algorithms in these fields rely on information theoretic approaches such as [[entropy]], [[mutual information]], or [[Kullback–Leibler divergence]]. However, to estimate these quantities, one must first either perform density estimation, or employ sophisticated space-partitioning/bias-correction strategies which are typically infeasible for high-dimensional data.<ref name = "SongThesis">L. Song. (2008) [http://www.cc.gatech.edu/~lsong/papers/lesong_thesis.pdf Learning via Hilbert Space Embedding of Distributions]. PhD Thesis, University of Sydney.</ref> Commonly, methods for modeling complex distributions rely on parametric assumptions that may be unfounded or computationally challenging (e.g. [[Mixture model#Gaussian mixture model|Gaussian mixture models]]), while nonparametric methods like [[kernel density estimation]] (Note: the smoothing kernels in this context have a different interpretation than the kernels discussed here) or [[characteristic function (probability theory)|characteristic function]] representation (via the [[Fourier transform]] of the distribution) break down in high-dimensional settings.<ref name = "Song2013" />▼
▲The analysis of distributions is fundamental in [[machine learning]] and [[statistics]], and many algorithms in these fields rely on information theoretic approaches such as [[entropy]], [[mutual information]], or [[Kullback–Leibler divergence]]. However, to estimate these quantities, one must first either perform density estimation, or employ sophisticated space-partitioning/bias-correction strategies which are typically infeasible for high-dimensional data.<ref name = "SongThesis">L. Song. (2008) [
Methods based on the kernel embedding of distributions sidestep these problems and also possess the following advantages:<ref name = "SongThesis" />
Line 14 ⟶ 17:
==Definitions==
Let <math>X</math> denote a random variable with
:<math>\
One may alternatively consider <math>x \mapsto k(x,\cdot)</math> as an implicit feature mapping <math>\
===Kernel embedding===
The kernel embedding of the distribution <math>P
:<math>\mu_X := \mathbb{E}
If <math>P</math> allows a square integrable density <math>p</math>, then <math>\mu_X = \mathcal{E}_k p</math>, where <math>\mathcal{E}_k</math> is the [[Hilbert–Schmidt integral operator]]. A kernel is ''characteristic'' if the mean embedding <math>\mu: \{\text{family of distributions over }\Omega \} \to \mathcal{H} </math> is injective.<ref name = "Fukumizu2008">K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf (2008). [http://papers.nips.cc/paper/3340-kernel-measures-of-conditional-dependence.pdf Kernel measures of conditional independence]. ''Advances in Neural Information Processing Systems'' '''20''', MIT Press, Cambridge, MA.</ref> Each distribution can thus be uniquely represented in the RKHS and all statistical features of distributions are preserved by the kernel embedding if a characteristic kernel is used.
===Empirical kernel embedding===
Given <math>n</math> training examples <math>\{x_1, \
:<math>\widehat{\mu}_X = \frac{1}{n} \sum_{i=1}^n \
===Joint distribution embedding===
If <math>Y</math> denotes another random variable (for simplicity, assume the co-___domain of <math>Y</math> is also <math>\Omega</math> with the same kernel <math>k</math> which satisfies <math> \langle \
:<math> \mathcal{C}_{XY} = \mathbb{E
By the equivalence between a [[tensor]] and a [[linear map]], this joint embedding may be interpreted as an uncentered [[cross-covariance]] operator <math>\mathcal{C}_{XY}: \mathcal{H} \to \mathcal{H}</math> from which the cross-covariance of
:<math>\
Given <math>n</math> pairs of training examples <math>\{(x_1, y_1), \dots, (x_n, y_n)\} </math> drawn i.i.d. from <math>P</math>, we can also empirically estimate the joint distribution kernel embedding via
:<math>\widehat{\mathcal{C}}_{XY} = \frac{1}{n} \sum_{i=1}^n \
===Conditional distribution embedding===
Given a [[conditional distribution]] <math>P(
:<math>\mu_{Y
Note that the embedding of <math>P(
:<math>\begin{cases} \mathcal{C}_{Y
which given the feature mapping of <math>x</math> outputs the conditional embedding of <math>Y</math> given <math>X = x.</math> Assuming that for all <math>g \in \mathcal{H}: \mathbb{E
:<math> \mu_{Y
This assumption is always true for finite domains with characteristic kernels, but may not necessarily hold for continuous domains.<ref name = "Song2013"/> Nevertheless, even in cases where the assumption fails, <math> \mathcal{C}_{Y
Given training examples <math>\{(x_1, y_1),\dots, (x_n, y_n)\},</math> the empirical kernel conditional embedding operator may be estimated as <ref name = "Song2013" />
:<math>\widehat{C}_{Y
where <math>\boldsymbol{\Phi} = \left(\
Thus, the empirical estimate of the kernel conditional embedding is given by a weighted sum of samples of <math>Y</math> in the feature space:
:<math> \widehat{\mu}_{Y
where <math> \boldsymbol{\beta}(x) = (\mathbf{K} + \lambda \mathbf{I})^{-1} \mathbf{K}_x</math> and <math> \mathbf{K}_x = \left( k(x_1, x), \dots, k(x_n, x) \right)^T </math>
Line 75 ⟶ 78:
* The expectation of any function <math> f </math> in the RKHS can be computed as an inner product with the kernel embedding:
::<math> \mathbb{E}
* In the presence of large sample sizes, manipulations of the <math>n \times n</math> Gram matrix may be computationally demanding. Through use of a low-rank approximation of the Gram matrix (such as the [[incomplete Cholesky factorization]]), running time and memory requirements of kernel-embedding-based learning algorithms can be drastically reduced without suffering much loss in approximation accuracy.<ref name = "Song2013"/>
Line 82 ⟶ 85:
* If <math>k</math> is defined such that <math>f</math> takes values in <math>[0,1]</math> for all <math>f \in \mathcal{H}</math> with <math>\| f\|_\mathcal{H} \le 1 </math> (as is the case for the widely used [[radial basis function]] kernels), then with probability at least <math>1-\delta </math>:<ref name="SongThesis" />
::<math>\|\mu_X - \widehat{\mu}_X \|_\mathcal{H} = \sup_{f \in \mathcal{B}(0,1)} \left| \mathbb{E}
:where <math>\mathcal{B}(0,1)</math> denotes the unit ball in <math>\mathcal{H}</math> and <math>\mathbf{K} =(k_{ij})</math> is the Gram matrix with <math>k_{ij} = k(x_i, x_j).</math>
* The rate of convergence (in RKHS norm) of the empirical kernel embedding to its distribution counterpart is <math>O(n^{-1/2})</math> and does ''not'' depend on the dimension of <math>X</math>. ▼
▲* The rate of convergence (in RKHS norm) of the empirical kernel embedding to its distribution counterpart is <math>O(n^{-1/2})</math> and does ''not'' depend on the dimension of <math>X</math>.
* Statistics based on kernel embeddings thus avoid the [[curse of dimensionality]], and though the true underlying distribution is unknown in practice, one can (with high probability) obtain an approximation within <math>O(n^{-1/2})</math> of the true kernel embedding based on a finite sample of size <math>n</math>.
* For the embedding of conditional distributions, the empirical estimate can be seen as a ''weighted'' average of feature mappings (where the weights <math>\beta_i(x) </math> depend on the value of the conditioning variable and capture the effect of the conditioning on the kernel embedding). In this case, the empirical estimate converges to the conditional distribution RKHS embedding with rate <math>O\left(n^{-1/4} \right)</math> if the regularization parameter <math>\lambda</math> is decreased as <math>O\left( n^{-1/2} \right),</math> though faster rates of convergence may be achieved by placing additional assumptions on the joint distribution.<ref name="Song2013"/>
=== Universal kernels ===
* Let <math>\mathcal{X} \subseteq \mathbb{R}^{b}</math> be a compact metric space and <math>C(\mathcal{X})</math> the set of [[Function space#Functional analysis|continuous functions]]. The reproducing kernel <math>k:\mathcal{X}\times \mathcal{X} \rightarrow \mathbb{R} </math> is called '''universal''' if and only if the RKHS <math>\mathcal{H}</math> of <math>k</math> is [[Dense set|dense]] in <math>C(\mathcal{X})</math>, i.e., for any <math>g \in C(\mathcal{X})</math> and all <math>\varepsilon > 0</math> there exists an <math> f \in \mathcal{H}</math> such that <math> \| f-g\|_{\infty} \leq \varepsilon</math>.<ref>*{{cite book |last1=Steinwart |first1=Ingo |last2=Christmann |first2=Andreas |title=Support Vector Machines |publisher=Springer |___location=New York |year=2008 |isbn=978-0-387-77241-7 }}</ref> All universal kernels defined on a compact space are characteristic kernels but the converse is not always true.<ref>{{Cite journal|last1=Sriperumbudur|first1= B. K.|last2=Fukumizu|first2=K.|last3=Lanckriet|first3=G.R.G.|title=Universality, Characteristic Kernels and RKHS Embedding of Measures|year= 2011 |journal=Journal of Machine Learning Research|volume=12|number=70}}</ref>
*
::<math>h(t) = \
:For <math>k</math> to be universal it suffices that the continuous part of <math>\mu</math> in its unique [[Lebesgue's decomposition theorem|Lebesgue decomposition]] <math>\mu = \mu_c + \mu_s</math> is non-zero. Furthermore, if
::<math>d\mu_c(\omega) = s(\omega)d\omega,</math>
:then <math>s</math> is the [[spectral density]] of frequencies <math>\omega</math> in <math>\mathbb{R}^{b}</math> and <math>h</math> is the [[Fourier transform]] of <math>s</math>. If the [[Support (mathematics)|support]] of <math>\mu</math> is all of <math>\mathbb{R}^{b}</math>, then <math>k</math> is a characteristic kernel as well.<ref>{{Citation |last=Liang|first=Percy| year=2016|title=CS229T/STAT231: Statistical Learning Theory | series = Stanford lecture notes | url=https://web.stanford.edu/class/cs229t/notes.pdf}}</ref><ref>{{cite conference|last1=Sriperumbudur|first1= B. K.|last2=Fukumizu|first2=K.|last3=Lanckriet|first3=G.R.G.|title=On the relation between universality, characteristic kernels and RKHS embedding of measures|url= https://proceedings.mlr.press/v9/sriperumbudur10a.html|year=2010 | conference=Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics|___location=Italy}}
</ref><ref>{{cite journal | last1 = Micchelli|first1=C.A.| last2 = Xu|first2=Y.| last3 = Zhang|first3=H.| title = Universal Kernels | journal = Journal of Machine Learning Research | year = 2006 | volume = 7 | number = 95| pages = 2651–2667| url= http://jmlr.org/papers/v7/micchelli06a.html }}</ref>
* If <math>k</math> induces a strictly positive definite kernel matrix for any set of distinct points, then it is a universal kernel.<ref name = "SongThesis" /> For example, the widely used Gaussian RBF kernel
::<math> k(x,x') = \exp\left(-\frac{1}{2\sigma^2} \|x-x'\|^2 \right)</math>
:on compact subsets of <math>\mathbb{R}^
▲::<math>h(t) = \int e^{-i\langle t, \omega \rangle} \mu(d\omega)</math>
=== Parameter selection for conditional distribution kernel embeddings ===
* The empirical kernel conditional distribution embedding operator <math>\widehat{\mathcal{C}}_{Y|X}</math> can alternatively be viewed as the solution of the following regularized least squares (function-valued) regression problem <ref>S. Grunewalder, G. Lever, L. Baldassarre, S. Patterson, A. Gretton, M. Pontil. (2012). [http://icml.cc/2012/papers/898.pdf Conditional mean embeddings as regressors]. ''Proc. Int. Conf. Machine Learning'': 1823–1830.</ref>
::<math>\min_{\mathcal{C}: \mathcal{H} \to \mathcal{H}} \sum_{i=1}^n \left \|\
:where <math>\|\cdot\|_{HS}</math> is the [[Hilbert–Schmidt operator|
* One can thus select the regularization parameter <math>\lambda</math> by performing [[cross-validation (statistics)|cross-validation]] based on the squared loss function of the regression problem.
Line 115:
== Rules of probability as operations in the RKHS ==
This section illustrates how basic probabilistic rules may be reformulated as (multi)linear algebraic operations in the kernel embedding framework and is primarily based on the work of Song et al.<ref name = "Song2013" /><ref name = "SongCDE" /> The following notation is adopted:
* <math>P(X,Y)= </math> joint distribution over random variables <math> X, Y </math>
* <math>P(X)= \int_\Omega P(X, \mathrm{d}y) = </math> marginal distribution of <math>X</math>; <math>P(Y)= </math> marginal distribution of <math>Y </math>
* <math> P(Y
* <math> \pi(Y) = </math> prior distribution over <math> Y </math>
* <math> Q </math> is used to distinguish distributions which incorporate the prior from distributions <math> P </math> which do not rely on the prior
Line 132 ⟶ 129:
In probability theory, the marginal distribution of <math>X</math> can be computed by integrating out <math> Y </math> from the joint density (including the prior distribution on <math>Y</math>)
:<math> Q(X) = \int_\Omega P(X
The analog of this rule in the kernel embedding framework states that <math>\mu_X^\pi,</math> the RKHS embedding of <math>Q(X)</math>, can be computed via
:<math>\mu_X^\pi = \mathbb{E
where <math>\mu_Y^\pi</math> is the kernel embedding of <math>\pi(Y).</math> In practical implementations, the kernel sum rule takes the following form
:<math> \widehat{\mu}_X^\pi = \widehat{\mathcal{C}}_{X
where
:<math>\mu_Y^\pi = \sum_{i=1}^{\widetilde{n}} \alpha_i \
is the empirical kernel embedding of the prior distribution, <math>\boldsymbol{\alpha} = (\alpha_1, \ldots, \alpha_{\widetilde{n}} )^T,</math> <math>\boldsymbol{\Upsilon} = \left(\
=== Kernel chain rule ===
In probability theory, a joint distribution can be factorized into a product between conditional and marginal distributions
:<math>Q(X,Y) = P(X
The analog of this rule in the kernel embedding framework states that <math> \mathcal{C}_{XY}^\pi,</math> the joint embedding of <math>Q(X,Y),</math> can be factorized as a composition of conditional embedding operator with the auto-covariance operator associated with <math>\pi(Y)</math>
:<math>\mathcal{C}_{XY}^\pi = \mathcal{C}_{X
where
:<math>\mathcal{C}_{XY}^\pi = \mathbb{E
:<math>\mathcal{C}_{YY}^\pi = \mathbb{E}
In practical implementations, the kernel chain rule takes the following form
:<math> \widehat{\mathcal{C}}_{XY}^\pi = \widehat{\mathcal{C}}_{X
=== Kernel Bayes' rule ===
In probability theory, a posterior distribution can be expressed in terms of a prior distribution and a likelihood function as
:<math>Q(Y
The analog of this rule in the kernel embedding framework expresses the kernel embedding of the conditional distribution in terms of conditional embedding operators which are modified by the prior distribution
:<math> \mu_{Y
where from the chain rule:
:<math> \mathcal{C}_{YX}^\pi = \left( \mathcal{C}_{X
In practical implementations, the kernel Bayes' rule takes the following form
:<math>\widehat{\mu}_{Y
where
:<math>\boldsymbol{\Lambda} = \left(\mathbf{G} + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widetilde{\mathbf{G}} \
Two regularization parameters are used in this framework: <math>\lambda </math> for the estimation of <math> \widehat{\mathcal{C}}_{YX}^\pi, \widehat{\mathcal{C}}_{XX}^\pi = \boldsymbol{\Upsilon} \mathbf{D} \boldsymbol{\Upsilon}^T</math> and <math>\widetilde{\lambda}</math> for the estimation of the final conditional embedding operator
:<math>\widehat{\mathcal{C}}_{Y
The latter regularization is done on square of <math>\widehat{\mathcal{C}}_{XX}^\pi</math> because <math>D</math> may not be [[Positive-definite matrix|positive definite]].
Line 196 ⟶ 193:
=== Measuring distance between distributions ===
The '''maximum mean discrepancy
:<math>\text{MMD}(P,Q) = \left \| \mu_X - \mu_Y \right \|_{\mathcal{H}}
While most distance-measures between distributions such as the widely used [[Kullback–Leibler divergence]] either require density estimation (either parametrically or nonparametrically) or space partitioning/bias correction strategies,<ref name = "SongThesis" /> the MMD is easily estimated as an empirical mean which is concentrated around the true value of the MMD. The characterization of this distance as the ''maximum mean discrepancy'' refers to the fact that computing the MMD is equivalent to finding the RKHS function that maximizes the difference in expectations between the two probability distributions
:<math>\text{MMD}(P,Q) = \sup_{\|f \|_\mathcal{H} \le 1} \left( \mathbb{E}
a form of [[integral probability metric]].
▲:<math>\text{MMD}(P,Q) = \sup_{\|f \|_\mathcal{H} \le 1} \left( \mathbb{E}_X[f(X)] - \mathbb{E}_Y[f(Y)] \right)</math>
=== Kernel two-sample test ===
Given ''n'' training examples from <math>P(X)</math> and ''m'' samples from <math>Q(Y)</math>, one can formulate a test statistic based on the empirical estimate of the MMD
:<math>
:<math>\widehat{\text{MMD}}(P,Q) = \left\| \frac{1}{n}\sum_{i=1}^n \phi(x_i) - \frac{1}{m}\sum_{i=1}^m \phi(y_i) \right \|_{\mathcal{H}}^2 = \frac{1}{n^2} \sum_{i=1}^n\sum_{j=1}^n k(x_i, x_j) + \frac{1}{m^2} \sum_{i=1}^m\sum_{j=1}^m k(y_i, y_j) - \frac{2}{nm} \sum_{i=1}^n\sum_{j=1}^m k(x_i, y_j) </math>▼
\begin{align}
\widehat{\text{MMD}}(P,Q) & = \left\| \frac{1}{n}\sum_{i=1}^n \varphi(x_i) - \frac{1}{m}\sum_{i=1}^m \varphi(y_i) \right \|_{\mathcal{H}}^2 \\[5pt]
▲
\end{align}
</math>
to obtain a '''two-sample test''' <ref>A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, A. Smola. (2012). [http://jmlr.org/papers/volume13/gretton12a/gretton12a.pdf A kernel two-sample test]. ''Journal of Machine Learning Research'', '''13''':
=== Density estimation via kernel embeddings ===
Although learning algorithms in the kernel embedding framework circumvent the need for intermediate density estimation, one may nonetheless use the empirical embedding to perform density estimation based on ''n'' samples drawn from an underlying distribution <math>P_X^*</math>. This can be done by solving the following optimization problem <ref name ="SongThesis"/><ref>M. Dudík, S. J. Phillips, R. E. Schapire. (2007). [http://classes.soe.ucsc.edu/cmps242/Winter08/lect/15/maxent_genreg_jmlr.pdf Maximum Entropy Distribution Estimation with Generalized Regularization and an Application to Species Distribution Modeling]. ''Journal of Machine Learning Research'', '''8''':
:<math> \max_{P_X} H(P_X) </math> subject to <math>\|\widehat{\mu}_X - \mu_X[P_X] \|_\mathcal{H} \le \
where the maximization is done over the entire space of distributions on <math>\Omega.</math> Here, <math>\mu_X[P_X]</math> is the kernel embedding of the proposed density <math>P_X</math> and <math>H</math> is an entropy-like quantity (e.g. [[Entropy (information theory)|Entropy]], [[Kullback–Leibler divergence|KL divergence]], [[Bregman divergence]]). The distribution which solves this optimization may be interpreted as a compromise between fitting the empirical kernel means of the samples well, while still allocating a substantial portion of the probability mass to all regions of the probability space (much of which may not be represented in the training examples). In practice, a good approximate solution of the difficult optimization may be found by restricting the space of candidate densities to a mixture of ''M'' candidate distributions with regularized mixing proportions. Connections between the ideas underlying [[Gaussian process]]es and [[conditional random fields]] may be drawn with the estimation of conditional probability distributions in this fashion, if one views the feature mappings associated with the kernel as sufficient statistics in generalized (possibly infinite-dimensional) [[exponential family|exponential families]].<ref name = "SongThesis"/>
=== Measuring dependence of random variables ===
A measure of the statistical dependence between random variables <math>X</math> and <math>Y</math> (from any domains on which sensible kernels can be defined) can be formulated based on the Hilbert–Schmidt Independence Criterion <ref>A. Gretton, O. Bousquet, A. Smola, B. Schölkopf. (2005). [http://www.gatsby.ucl.ac.uk/~gretton/papers/GreBouSmoSch05.pdf Measuring statistical dependence with Hilbert–Schmidt norms]. ''Proc. Intl. Conf. on Algorithmic Learning Theory'': 63–78.</ref>
:<math> \text{HSIC}(X, Y) = \left \| \mathcal{C}_{XY} - \mu_X \otimes \mu_Y \right \|_{\mathcal{H} \otimes \mathcal{H}}^2 </math>
and can be used as a principled replacement for [[mutual information]], [[Pearson correlation]] or any other dependence measure used in learning algorithms. Most notably, HSIC can detect arbitrary dependencies (when a characteristic kernel is used in the embeddings, HSIC is zero if and only if the variables are [[independence (probability theory)|independent]]), and can be used to measure dependence between different types of data (e.g. images and text captions). Given ''n'' i.i.d. samples of each random variable, a simple parameter-free [[Bias of an estimator|unbiased]] estimator of HSIC which exhibits [[Concentration of measure|concentration]] about the true value can be computed in <math>O(n(d_f^2 +d_g^2))</math> time,<ref name = "SongThesis"/> where the Gram matrices of the two datasets are approximated using <math>\mathbf{A} \mathbf{A}^T, \mathbf{B} \mathbf{B}^T </math> with <math>\mathbf{A} \in \R^{n \times d_f}, \mathbf{B} \in \R^{n \times d_g}</math>. The desirable properties of HSIC have led to the formulation of numerous algorithms which utilize this dependence measure for a variety of common machine learning tasks such as: [[feature selection]] (BAHSIC <ref>L. Song, A. Smola
HSIC can be extended to measure the dependence of multiple random variables. The question of when HSIC captures independence in this case has recently been studied:<ref name = "CharAndUniv">Zoltán Szabó, Bharath K. Sriperumbudur. [http://jmlr.org/papers/v18/17-492.html Characteristic and Universal Tensor Product Kernels]. ''Journal of Machine Learning Research'', 19:
more than two variables
* on <math>\R^d</math>: the characteristic property of the individual kernels remains an equivalent condition.
Line 231 ⟶ 233:
=== Kernel belief propagation ===
[[Belief propagation]] is a fundamental algorithm for inference in [[graphical
:<math>m_{ut}(\cdot) = \sum_{i=1}^n \beta_{ut}^i \
if it assumed to lie in the RKHS. The '''kernel belief propagation update''' message from ''t'' to node ''s'' is then given by <ref name = "Song2013"/>
:<math> \widehat{m}_{ts} = \left( \odot_{u \in N(t) \backslash s} \mathbf{K}_t \boldsymbol{\beta}_{ut} \right)^T (\mathbf{K}_s + \lambda \mathbf{I} )^{-1} \boldsymbol{\Upsilon}_s^T \
where <math>\odot</math> denotes the element-wise vector product, <math>N(t) \backslash s </math> is the set of nodes connected to ''t'' excluding node ''s'', <math> \boldsymbol{\beta}_{ut} = \left(\beta_{ut}^1, \dots, \beta_{ut}^n \right) </math>, <math>\mathbf{K}_t, \mathbf{K}_s </math> are the Gram matrices of the samples from variables <math>X_t, X_s </math>, respectively, and <math>\boldsymbol{\Upsilon}_s = \left(\
Thus, if the incoming messages to node ''t'' are linear combinations of feature mapped samples from <math> X_t </math>, then the outgoing message from this node is also a linear combination of feature mapped samples from <math> X_s </math>. This RKHS function representation of message-passing updates therefore produces an efficient belief propagation algorithm in which the [[Markov
=== Nonparametric filtering in hidden Markov models ===
In the [[hidden Markov model]] (HMM), two key quantities of interest are the transition probabilities between hidden states <math> P(S^t \mid S^{t-1})</math> and the emission probabilities <math>P(O^t \mid S^t)</math> for observations. Using the kernel conditional distribution embedding framework, these quantities may be expressed in terms of samples from the HMM. A serious limitation of the embedding methods in this ___domain is the need for training samples containing hidden states, as otherwise inference with arbitrary distributions in the HMM is not possible.
One common use of HMMs is [[Hidden Markov
:<math>\mu_{S^{t+1} \mid h^{t+1}} = \mathcal{C}_{S^{t+1} O^{t+1}}^\pi \left(\mathcal{C}_{O^{t+1} O^{t+1}}^\pi \right)^{-1} \
by computing the embeddings of the prediction step via the [[#Kernel
:<math>\widehat{\mu}_{S^{t+1} \mid h^{t+1}} = \sum_{i=1}^T \alpha_i^t \
and filtering with kernel embeddings is thus implemented recursively using the following updates for the weights <math>\boldsymbol{\alpha} = (\alpha_1, \dots, \alpha_T)</math> <ref name = "Song2013"/>
:<math>\mathbf{D}^{t+1} = \
:<math>\boldsymbol{\alpha}^{t+1} = \mathbf{D}^{t+1} \mathbf{K} \left( (\mathbf{D}^{t+1} K)^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \mathbf{D}^{t+1} \mathbf{K}_{o^{t+1}} </math>
Line 264 ⟶ 266:
The '''support measure machine''' (SMM) is a generalization of the [[support vector machine]] (SVM) in which the training examples are probability distributions paired with labels <math> \{P_i, y_i\}_{i=1}^n, \ y_i \in \{+1, -1\} </math>.<ref name = "SMM">K. Muandet, K. Fukumizu, F. Dinuzzo, B. Schölkopf. (2012). [http://books.nips.cc/papers/files/nips25/NIPS2012_0015.pdf Learning from Distributions via Support Measure Machines]. ''Advances in Neural Information Processing Systems'': 10–18.</ref> SMMs solve the standard SVM [[Support vector machine#Dual form|dual optimization problem]] using the following '''expected kernel'''
:<math> K\left(P(X), Q(Z)\right) = \langle \mu_X , \mu_Z \rangle_\mathcal{H} = \mathbb{E
which is computable in closed form for many common specific distributions <math> P_i </math> (such as the Gaussian distribution) combined with popular embedding kernels <math>k</math> (e.g. the Gaussian kernel or polynomial kernel), or can be accurately empirically estimated from i.i.d. samples <math>\{x_i\}_{i=1}^n \sim P(X), \{z_j\}_{j=1}^m \sim Q(Z) </math> via
Line 273 ⟶ 275:
=== Domain adaptation under covariate, target, and conditional shift ===
The goal of [[
# '''Covariate
# '''Target
# '''Conditional
By utilizing the kernel embedding of marginal and conditional distributions, practical approaches to deal with the presence of these types of differences between training and test domains can be formulated. Covariate shift may be accounted for by reweighting examples via estimates of the ratio <math>P^\text{te}(X)/P^\text{tr}(X)</math> obtained directly from the kernel embeddings of the marginal distributions of <math>X</math> in each ___domain without any need for explicit estimation of the distributions.<ref name = "CovS"/> Target shift, which cannot be similarly dealt with since no samples from <math>Y</math> are available in the test ___domain, is accounted for by weighting training examples using the vector <math>\boldsymbol{\beta}^*(\mathbf{y}^\text{tr}) </math> which solves the following optimization problem (where in practice, empirical approximations must be used) <ref name = "DA"/>
:<math>\min_{\boldsymbol{\beta}(y)} \left \|\mathcal{C}_{{(X \mid Y)}^\text{tr}} \mathbb{E
To deal with ___location scale conditional shift, one can perform a LS transformation of the training points to obtain new transformed training data <math> \mathbf{X}^\text{new} = \mathbf{X}^\text{tr} \odot \mathbf{W} + \mathbf{B}</math> (where <math>\odot</math> denotes the element-wise vector product). To ensure similar distributions between the new transformed training samples and the test data, <math>\mathbf{W},\mathbf{B}</math> are estimated by minimizing the following empirical kernel embedding distance <ref name = "DA"/>
:<math>\left \| \widehat{\mu}_{X^\text{new}} - \widehat{\mu}_{X^\text{te}} \right \|_{\mathcal{H}}^2 = \left \| \widehat{\mathcal{C}}_{(X \mid Y)^\text{new}} \widehat{\mu}_{Y^\text{tr}} - \widehat{\mu}_{X^\text{te}} \right \|_{\mathcal{H}}^2 </math>
In general, the kernel embedding methods for dealing with LS conditional shift and target shift may be combined to find a reweighted transformation of the training data which mimics the test distribution, and these methods may perform well even in the presence of conditional shifts other than ___location-scale changes.<ref name = "DA"/>
Line 290 ⟶ 292:
=== Domain generalization via invariant feature representation ===
Given ''N'' sets of training examples sampled i.i.d. from distributions <math>P^{(1)}(X,Y), P^{(2)}(X,Y), \ldots, P^{(N)}(X,Y)</math>, the goal of '''___domain generalization''' is to formulate learning algorithms which perform well on test examples sampled from a previously unseen ___domain <math>P^*(X,Y)</math> where no data from the test ___domain is available at training time. If conditional distributions <math>P(Y \mid X)</math> are assumed to be relatively similar across all domains, then a learner capable of ___domain generalization must estimate a functional relationship between the variables which is robust to changes in the marginals <math>P(X)</math>. Based on kernel embeddings of these distributions, Domain Invariant Component Analysis (DICA) is a method which determines the transformation of the training data that minimizes the difference between marginal distributions while preserving a common conditional distribution shared between all training domains.<ref name = "DICA">K. Muandet, D. Balduzzi, B. Schölkopf. (2013).[http://jmlr.org/proceedings/papers/v28/muandet13.pdf Domain Generalization Via Invariant Feature Representation]. ''30th International Conference on Machine Learning''.</ref> DICA thus extracts ''invariants'', features that transfer across domains, and may be viewed as a generalization of many popular dimension-reduction methods such as [[kernel principal component analysis]], transfer component analysis, and covariance operator inverse regression.<ref name = "DICA"/>
Defining a probability distribution <math>\mathcal{P}</math> on the RKHS <math>\mathcal{H}</math> with ▼
▲Defining a probability distribution <math>\mathcal{P}</math> on the RKHS <math>\mathcal{H}</math> with
:<math>\mathcal{P} \left (\mu_{X^{(i)}Y^{(i)}} \right ) = \frac{1}{N} \qquad \text{ for } i=1,\dots, N,</math> ▼
▲:<math>\mathcal{P} \left (\mu_{X^{(i)}Y^{(i)}} \right ) = \frac{1}{N} \qquad \text{ for } i=1,\dots, N,</math>
DICA measures dissimilarity between domains via '''distributional variance''' which is computed as
:<math>V_\mathcal{H} (\mathcal{P}) = \frac{1}{N} \
where
:<math>\mathbf{G}_{ij} = \left \langle \mu_{X^{(i)}}, \mu_{X^{(j)}} \right \rangle_\mathcal{H} </math>
so <math>\mathbf{G}</math> is a <math>N \times N</math> Gram matrix over the distributions from which the training data are sampled. Finding an [[Orthogonal matrix|orthogonal transform]] onto a low-dimensional [[Linear subspace|subspace]] ''B'' (in the feature space) which minimizes the distributional variance, DICA simultaneously ensures that ''B'' aligns with the [[Basis function|bases]] of a '''central subspace''' ''C'' for which <math>Y</math> becomes independent of <math>X</math> given <math>C^T X</math> across all domains. In the absence of target values <math>Y</math>, an unsupervised version of DICA may be formulated which finds a low-dimensional subspace that minimizes distributional variance while simultaneously maximizing the variance of <math>X</math> (in the feature space) across all domains (rather than preserving a central subspace).<ref name = "DICA"/>
=== Distribution regression ===
In distribution regression, the goal is to regress from probability distributions to reals (or vectors). Many important [[machine learning]] and statistical tasks fit into this framework, including [[Multiple-instance learning|multi-instance learning]], and [[point estimation]] problems without analytical solution (such as [[Hyperparameter (Bayesian statistics)|hyperparameter]] or [[entropy estimation]]). In practice only samples from sampled distributions are observable, and the estimates have to rely on similarities computed between ''sets of points''. Distribution regression has been successfully applied for example in supervised entropy learning, and aerosol prediction using multispectral satellite images.<ref name = "MERR">Z. Szabó, B. Sriperumbudur, B. Póczos, A. Gretton. [http://jmlr.org/papers/v17/14-510.html Learning Theory for Distribution Regression]. ''Journal of Machine Learning Research'', 17(152):
Given <math>{\left(\{X_{i,n}\}_{n=1}^{N_i}, y_i\right)}_{i=1}^
:<math>J(f) = \frac{1}{\ell} \sum_{i=1}^
where
:<math>\mu_{\hat{X}_i} = \int_
with a <math>k</math> kernel on the ___domain of <math>X_i</math>-s <math>(k:\Omega\times \Omega \to \R)</math>, <math>K</math> is a kernel on the embedded distributions, and <math>\mathcal{H}(K)</math> is the RKHS determined by <math>K</math>. Examples for <math>K</math> include the linear kernel <math>\left[ K(\mu_P,\mu_Q) = \langle\mu_P,\mu_Q\rangle_{\mathcal{H}(k)} \right] </math>, the Gaussian kernel <math> \left[ K(\mu_P,\mu_Q) = e^{-\left\|\mu_P-\mu_Q\right\|_{H(k)}^2/(2\sigma^2)} \right] </math>, the exponential kernel <math> \left[ K(\mu_P,\mu_Q) = e^{-\left\|\mu_P-\mu_Q\right\|_{H(k)}/(2\sigma^2)} \right] </math>, the Cauchy kernel <math> \left[ K(\mu_P,\mu_Q) = \left(1+ \left\|\mu_P-\mu_Q\right\|_{H(k)}^2/\sigma^2 \right)^{-1} \right] </math>, the generalized t-student kernel <math> \left[ K(\mu_P,\mu_Q) = \left(1+ \left\|\mu_P-\mu_Q\right\|_{H(k)}^{\sigma} \right)^{-1}, (\sigma \le 2) \right] </math>, or the inverse multiquadrics kernel <math> \left[ K(\mu_P,\mu_Q) = \left(\left\|\mu_P-\mu_Q\right\|_{H(k)}^2 + \sigma^2 \right)^{-\frac{1}{2}} \right] </math>.
Line 320 ⟶ 322:
The prediction on a new distribution <math>(\hat{X})</math> takes the simple, analytical form
:: <math> \hat{y}\big(\hat{X}\big) = \mathbf{k} [\mathbf{G} + \lambda \ell]^{-1}\mathbf{y}, </math>
where <math>\mathbf{k}=\big[K \big(\mu_{\hat{X}_i},\mu_{\hat{X}}\big)\big]\in \R^{1\times \ell}</math>, <math>\mathbf{G}=[G_{ij}]\in \R^{\ell\times \ell}</math>, <math>G_{ij} = K\big(\mu_{\hat{X}_i},\mu_{\hat{X}_j}\big)\in \R</math>, <math>\mathbf{y}=[y_1;
== Example ==
In this simple example, which is taken from Song et al.,<ref name = "Song2013"/> <math>X, Y</math> are assumed to be [[Probability distribution#Discrete probability distribution|discrete random variables]] which take values in the set <math>\{1,\ldots,K\} </math> and the kernel is chosen to be the [[Kronecker delta]] function, so <math>k(x,x') = \delta(x,x')</math>. The feature map corresponding to this kernel is the [[standard basis]] vector <math>\
:<math>\mu_X = \mathbb{E}
:<math>\mathcal{C}_{XY} = \mathbb{E
:<math>\mathcal{C}_{Y\mid X} = \mathcal{C}_{YX} \mathcal{C}_{XX}^{-1},</math>
is in this setting a conditional probability table
Line 336 ⟶ 338:
:<math>\mathcal{C}_{Y \mid X} = ( P(Y=s \mid X=t))_{s,t \in \{1,\dots,K\}}</math>
and
:<math>\mathcal{C}_{XX} =\begin{pmatrix} P(X=1) & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & P(X=K) \\ \end{pmatrix}</math>
Line 342 ⟶ 344:
Thus, the embeddings of the conditional distribution under a fixed value of <math>X</math> may be computed as
:<math>\mu_{Y \mid x} = \mathcal{C}_{Y \mid X} \
In this discrete-valued setting with the Kronecker delta kernel, the [[#Rules of probability as operations in the RKHS|kernel sum rule]] becomes
:<math>\underbrace{\begin{pmatrix}
The [[#Rules of probability as operations in the RKHS|kernel chain rule]] in this case is given by
:<math>\underbrace{\begin{pmatrix} \\
\end{pmatrix} }_{\mathcal{C}_{YY}^\pi} </math>
| |||