Kernel embedding of distributions: Difference between revisions

Revision as of 02:56, 17 September 2019

Revision as of 09:37, 17 September 2019

Line 96:

== Rules of probability as operations in the RKHS ==

This section illustrates how basic probabilistic rules may be reformulated as (multi)linear algebraic operations in the kernel embedding framework and is primarily based on the work of Song et al.<ref name = "Song2013" /><ref name = "SongCDE" /> The following notation is adopted:

* <math> P(X,Y) = </math> joint distribution over random variables <math> X, Y </math>

* <math> P(X,Y)= </math> joint distribution over random variables <math> X, Y </math>

* <math>P(X)= \int_\Omega P(X, \mathrm{d}y) = </math> marginal distribution of <math> X </math>; <math> P(Y) = </math> marginal distribution of <math> Y </math>

* <math> P(Y \mid| X) = \frac{P(X,Y)}{P(X)} = </math> conditional distribution of <math> Y </math> given <math> X </math> with corresponding conditional embedding operator <math> \mathcal{C}_{Y \mid| X}</math>

* <math> \pi(Y) = </math> prior distribution over <math> Y </math>

* <math> Q </math> is used to distinguish distributions which incorporate the prior from distributions <math> P </math> which do not rely on the prior

* <math> Q </math> is used to distinguish distributions which incorporate the prior from distributions <math> P </math> which do not rely on the prior

In practice, all embeddings are empirically estimated from data <math> \{(x_1,y_1),\dots, (x_n, y_n)\} </math> and it assumed that a set of samples <math>\{\widetilde{y}_1, \dotsldots, \widetilde{y}_{\widetilde{n}} \} </math> may be used to estimate the kernel embedding of the prior distribution <math> \pi(Y) </math>.

=== Kernel sum rule ===

In probability theory, the marginal distribution of <math> X </math> can be computed by integrating out <math> Y </math> from the joint density (including the prior distribution on <math> Y </math>)

:: <math> Q(X) = \int_\Omega P(X \mid Y) \mathrm{d} \pi(Y) </math>

:<math> Q(X) = \int_\Omega P(X | Y) \mathrm{d} \pi(Y) </math>

The analog of this rule in the kernel embedding framework states that <math> \mu_X^\pi ,</math>, the RKHS embedding of <math> Q(X)</math>, can be computed via

:: <math> \mu_X^\pi = \mathbb{E}_{Y} [\mathcal{C}_{X \mid| Y} \phi(Y) ] = \mathcal{C}_{X\mid| Y} \mathbb{E}_{Y} [\phi(Y)] = \mathcal{C}_{X\mid| Y} \mu_Y^\pi </math> where <math>\mu_Y^\pi</math> is the kernel embedding of <math> \pi(Y) </math>

In practical implementations, the kernel sum rule takes the following form

:: <math> \widehat{\mu}_X^\pi = \widehat{\mathcal{C}}_{X \mid Y} \widehat{\mu}_Y^\pi = \boldsymbol{\Upsilon} (\mathbf{G} + \lambda \mathbf{I})^{-1} \widetilde{\mathbf{G}} \boldsymbol{\alpha} </math>

where <math>\mu_Y^\pi</math> is the kernel embedding of <math>\pi(Y).</math> In practical implementations, the kernel sum rule takes the following form

where <math>\mu_Y^\pi = \sum_{i=1}^{\widetilde{n}} \alpha_i \phi(\widetilde{y}_i)</math> is the empirical kernel embedding of the prior distribution, <math>\boldsymbol{\alpha} = (\alpha_1, \dots, \alpha_{\widetilde{n}} )^T</math>, <math>\boldsymbol{\Upsilon} = \left(\phi(x_1), \dots, \phi(x_n) \right) </math>, and <math>\mathbf{G}, \widetilde{\mathbf{G}} </math> are Gram matrices with entries <math>\mathbf{G}_{ij} = k(y_i, y_j), \widetilde{\mathbf{G}}_{ij} = k(y_i, \widetilde{y}_j) </math> respectively.

:: <math> \widehat{\mu}_X^\pi = \widehat{\mathcal{C}}_{X \mid| Y} \widehat{\mu}_Y^\pi = \boldsymbol{\Upsilon} (\mathbf{G} + \lambda \mathbf{I})^{-1} \widetilde{\mathbf{G}} \boldsymbol{\alpha} </math>

where

:<math>\mu_Y^\pi = \sum_{i=1}^{\widetilde{n}} \alpha_i \phi(\widetilde{y}_i)</math>

where <math>\mu_Y^\pi = \sum_{i=1}^{\widetilde{n}} \alpha_i \phi(\widetilde{y}_i)</math> is the empirical kernel embedding of the prior distribution, <math>\boldsymbol{\alpha} = (\alpha_1, \dotsldots, \alpha_{\widetilde{n}} )^T,</math>, <math>\boldsymbol{\Upsilon} = \left(\phi(x_1), \dotsldots, \phi(x_n) \right) </math>, and <math>\mathbf{G}, \widetilde{\mathbf{G}} </math> are Gram matrices with entries <math>\mathbf{G}_{ij} = k(y_i, y_j), \widetilde{\mathbf{G}}_{ij} = k(y_i, \widetilde{y}_j) </math> respectively.

=== Kernel chain rule ===

In probability theory, a joint distribution can be factorized into a product between conditional and marginal distributions

:: <math> Q(X,Y) = P(X \mid Y) \pi(Y) </math>

:<math>Q(X,Y) = P(X | Y) \pi(Y) </math>

The analog of this rule in the kernel embedding framework states that <math> \mathcal{C}_{XY}^\pi ,</math>, the joint embedding of <math> Q(X,Y) ,</math>, can be factorized as a composition of conditional embedding operator with the auto-covariance operator associated with <math> \pi(Y) </math>

:: <math> \mathcal{C}_{XY}^\pi = \mathcal{C}_{X \mid Y} \mathcal{C}_{YY}^\pi </math> where <math> \mathcal{C}_{XY}^\pi = \mathbb{E}_{XY} [\phi(X) \otimes \phi(Y) ] </math> and <math> \mathcal{C}_{YY}^\pi = \mathbb{E}_Y [\phi(Y) \otimes \phi(Y)] </math>

:<math>\mathcal{C}_{XY}^\pi = \mathcal{C}_{X | Y} \mathcal{C}_{YY}^\pi </math>

where

:<math>\mathcal{C}_{XY}^\pi = \mathbb{E}_{XY} [\phi(X) \otimes \phi(Y) ],</math>

:<math>\mathcal{C}_{YY}^\pi = \mathbb{E}_Y [\phi(Y) \otimes \phi(Y)].</math>

In practical implementations, the kernel chain rule takes the following form

:: <math> \widehat{\mathcal{C}}_{XY}^\pi = \widehat{\mathcal{C}}_{X \mid| Y} \widehat{\mathcal{C}}_{YY}^\pi = \boldsymbol{\Upsilon} (\mathbf{G} + \lambda \mathbf{I})^{-1} \widetilde{\mathbf{G}} \text{diag}(\boldsymbol{\alpha}) \boldsymbol{\widetilde{\Phi}}^T </math>

=== Kernel Bayes' rule ===

In probability theory, a posterior distribution can be expressed in terms of a prior distribution and a likelihood function as

:: <math> Q(Y \mid x) = \frac{P(x \mid Y) \pi(Y)}{Q(x)} </math> where <math> Q(x) = \int_\Omega P(x \mid y) \mathrm{d} \pi(y) </math>

::<math>Q(Y|x) = \frac{P(x|Y) \pi(Y)}{Q(x)} </math> where <math> Q(Xx) = \int_\Omega P(Xx \mid| Yy) \mathrm{d} \pi(Yy) </math>

The analog of this rule in the kernel embedding framework expresses the kernel embedding of the conditional distribution in terms of conditional embedding operators which are modified by the prior distribution

:: <math> \mu_{Y \mid |x}^\pi = \mathcal{C}_{Y \mid| X}^\pi \phi(x) = \mathcal{C}_{YX}^\pi \left ( \mathcal{C}_{XX}^\pi \right )^{-1} \phi(x) </math> where from the chain rule: <math> \mathcal{C}_{YX}^\pi = \left( \mathcal{C}_{X \mid Y} \mathcal{C}_{YY}^\pi \right)^T </math>.

where from the chain rule:

:<math> \mathcal{C}_{YX}^\pi = \left( \mathcal{C}_{X|Y} \mathcal{C}_{YY}^\pi \right)^T.</math>

In practical implementations, the kernel Bayes' rule takes the following form

:: <math> \widehat{\mu}_{Y \mid x}^\pi = \widehat{\mathcal{C}}_{YX}^\pi \left( (\widehat{\mathcal{C}}_{XX})^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widehat{\mathcal{C}}_{XX}^\pi \phi(x) = \widetilde{\boldsymbol{\Phi}} \boldsymbol{\Lambda}^T \left( (\mathbf{D} \mathbf{K})^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \mathbf{K} \mathbf{D} \mathbf{K}_x </math>

where :<math> \boldsymbolwidehat{\Lambdamu}_{Y|x}^\pi = \widehat{\mathcal{C}}_{YX}^\pi \left( \mathbfleft (\widehat{G\mathcal{C}}_{XX} \right )^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widetildewidehat{\mathbfmathcal{GC}} \text_{diagXX}^\pi \phi(x) = \widetilde{\boldsymbol{\alphaPhi}}), \mathbfboldsymbol{D\Lambda}^T = \text{diag}\left(\left (\mathbf{GD} \mathbf{K})^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widetildemathbf{K} \mathbf{G}D} \boldsymbolmathbf{\alphaK} \right) _x </math>.

Two regularization parameters are used in this framework: <math>\lambda </math> for the estimation of <math> \widehat{\mathcal{C}}_{YX}^\pi, \widehat{\mathcal{C}}_{XX}^\pi = \boldsymbol{\Upsilon} \mathbf{D} \boldsymbol{\Upsilon}^T </math> and <math>\widetilde{\lambda} </math> for the estimation of the final conditional embedding operator <math> \widehat{\mathcal{C}}_{Y \mid X}^\pi = \widehat{\mathcal{C}}_{YX}^\pi \left( (\widehat{\mathcal{C}}_{XX}^\pi )^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widehat{\mathcal{C}}_{XX}^\pi </math>. The latter regularization is done on square of <math> \widehat{\mathcal{C}}_{XX}^\pi </math> because <math> D </math> may not be [[Positive-definite matrix|positive definite]].

where

:<math>\boldsymbol{\Lambda} = \left(\mathbf{G} + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widetilde{\mathbf{G}} \text{diag}(\boldsymbol{\alpha}), \qquad \mathbf{D} = \text{diag}\left(\left(\mathbf{G} + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widetilde{\mathbf{G}} \boldsymbol{\alpha} \right).</math>

Two regularization parameters are used in this framework: <math>\lambda </math> for the estimation of <math> \widehat{\mathcal{C}}_{YX}^\pi, \widehat{\mathcal{C}}_{XX}^\pi = \boldsymbol{\Upsilon} \mathbf{D} \boldsymbol{\Upsilon}^T </math> and <math>\widetilde{\lambda} </math> for the estimation of the final conditional embedding operator <math> \widehat{\mathcal{C}}_{Y \mid X}^\pi = \widehat{\mathcal{C}}_{YX}^\pi \left( (\widehat{\mathcal{C}}_{XX}^\pi )^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widehat{\mathcal{C}}_{XX}^\pi </math>. The latter regularization is done on square of <math> \widehat{\mathcal{C}}_{XX}^\pi </math> because <math> D </math> may not be [[Positive-definite matrix|positive definite]].

:: <math> \widehat{\mumathcal{C}}_{Y \mid x|X}^\pi = \widehat{\mathcal{C}}_{YX}^\pi \left( \left (\widehat{\mathcal{C}}_{XX}^\pi \right )^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \widehat{\mathcal{C}}_{XX}^\pi \phi(x) = \widetilde{\boldsymbol{\Phi}} \boldsymbol{\Lambda}^T \left( (\mathbf{D} \mathbf{K})^2 + \widetilde{\lambda} \mathbf{I} \right)^{-1} \mathbf{K} \mathbf{D} \mathbf{K}_x .</math>

The latter regularization is done on square of <math>\widehat{\mathcal{C}}_{XX}^\pi</math> because <math>D</math> may not be [[Positive-definite matrix|positive definite]].

==Applications==