Content deleted Content added
→See also: describing multimodal transformers |
Moved up the section on multimodal transformers and changed the titles |
||
Line 12:
Many models and algorithms have been implemented to retrieve and classify certain types of data, e.g. image or text (where humans who interact with machines can extract images in form of pictures and texts that could be any message etc.). However, data usually come with different modalities (it is the degree to which a system's components may be separated or combined) which carry different information. For example, it is very common to caption an image to convey the information not presented in the image itself. Similarly, sometimes it is more straightforward to use an image to describe the information which may not be obvious from texts. As a result, if different words appear in similar images, then these words likely describe the same thing. Conversely, if a word is used to describe seemingly dissimilar images, then these images may represent the same object. Thus, in cases dealing with multi-modal data, it is important to use a model which is able to jointly represent the information such that the model can capture the correlation structure between different modalities. Moreover, it should also be able to recover missing modalities given observed ones (e.g. predicting possible image object according to text description). The '''Multimodal Deep Boltzmann Machine model''' satisfies the above purposes.
== Multimodal transformers ==▼
==Background: Boltzmann machine==▼
{{excerpt|Transformer (machine learning model)|Multimodality}}▼
A [[Boltzmann machine]] is a type of stochastic neural network invented by [[Geoffrey Hinton]] and [[Terry Sejnowski]] in 1985. Boltzmann machines can be seen as the [[stochastic process|stochastic]], [[generative model|generative]] counterpart of [[Hopfield net]]s. They are named after the [[Boltzmann distribution]] in statistical mechanics. The units in Boltzmann machines are divided into two groups: visible units and hidden units. General Boltzmann machines allow connection between any units. However, learning is impractical using general Boltzmann Machines because the computational time is exponential to the size of the machine{{Citation needed|date=November 2022}}. A more efficient architecture is called '''[[restricted Boltzmann machine]]''' where connection is only allowed between hidden unit and visible unit, which is described in the next section.▼
=== Background ===
▲A [[Boltzmann machine]] is a type of stochastic neural network invented by [[Geoffrey Hinton]] and [[Terry Sejnowski]] in 1985. Boltzmann machines can be seen as the [[stochastic process|stochastic]], [[generative model|generative]] counterpart of [[Hopfield net]]s. They are named after the [[Boltzmann distribution]] in statistical mechanics. The units in Boltzmann machines are divided into two groups: visible units and hidden units. General Boltzmann machines allow connection between any units. However, learning is impractical using general Boltzmann Machines because the computational time is exponential to the size of the machine{{Citation needed|date=November 2022}}. A more efficient architecture is called
==== Restricted Boltzmann machine ====
A restricted Boltzmann machine<ref>{{cite web|url=https://www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.2014/pdfs/Smolensky.1986.pdf|title=Restricted Boltzmann Machine|year=1986|access-date=2019-08-29|archive-date=2016-03-03|archive-url=https://web.archive.org/web/20160303223045/http://www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.2014/pdfs/Smolensky.1986.pdf|url-status=live}}</ref> is an undirected [[Graph (discrete mathematics)|graph]] model with stochastic visible variables and stochastic hidden variables. Each visible variable is connected to each hidden variable. The energy function of the model is defined as
:<math> E(\mathbf v,\mathbf h;\theta) = -\sum_{i=1}^D\sum_{j=1}^{F}W_{ij}v_ih_j -\sum_{i=1}^Db_iv_i -\sum_{j=1}^Fa_jh_j</math>
Line 29 ⟶ 34:
The derivative of the log-likelihood with respect to the model parameters can be decomposed as the difference between the ''model's expectation'' and ''data-dependent expectation''.
==== Gaussian-Bernoulli RBM ====
:<math> E(\mathbf v,\mathbf h;\theta) = \sum_{i=1}^D\frac{(v_i-b_i)^2}{2\sigma_i^2} -\sum_{i=1}^D\sum_{j=1}^{F}\frac{v_i}{\sigma_i}W_{ij}v_ih_j -\sum_{i=1}^Db_iv_i -\sum_{j=1}^Fa_jh_j</math>
where <math>\theta = \{\mathbf a,\mathbf b,\mathbf w,\mathbf \sigma\}</math> are the model parameters. The joint distribution is defined the same as the one in [[#Restricted Boltzmann machine|restricted Boltzmann machine]]. The conditional distributions now become
Line 37 ⟶ 42:
In Gaussian-Bernoulli RBM, the visible unit conditioned on hidden units is modeled as a Gaussian distribution.
==== Replicated Softmax Model ====
The
:<math>E(\mathbf V,\mathbf h) = -\sum_{j=1}^{F}\sum_{k=1}^{K}W_{jk}\hat v_kh_j - \sum_{k=1}^Kb_k\hat v_k - M\sum_{j=1}^{F}a_jh_j</math>
The conditional distributions are given by
Line 44 ⟶ 49:
:<math>p(v_{ik} = 1|\mathbf h) = \frac{\mathrm{exp}(b_k + \sum_{j=1}^Fh_jW_{jk}}{\sum_{q=1}^{K}\mathrm{exp}(b_q + \sum_{j=1}^Fh_jW_{jq}})</math>
=== Deep Boltzmann machines ===
A
<math>
\begin{align}
Line 57 ⟶ 62:
:<math>P(\mathbf{v};\theta) = \frac{1}{\mathcal{Z}(\theta)}\sum_{\mathbf h}\mathrm{exp}(-E(\mathbf v,\mathbf h^{(1)},\mathbf h^{(2)},\mathbf h^{(3)};\theta))</math>
=== Multimodal deep Boltzmann machines ===
<math>
\begin{align}
Line 79 ⟶ 84:
:<math>p(v_i^m|\mathbf h^{(1m)}) \sim \mathcal{N}(\sigma_i\sum_{j=1}^{F_1^m}W_{ij}^{(1m)}h_j^{(1m)} + b_i^m,\sigma_i^2)</math>
==== Inference and learning ====
Exact maximum likelihood learning in this model is intractable, but approximate learning of DBMs can be carried out by using a variational approach, where mean-field inference is used to estimate data-dependent expectations and an MCMC based stochastic approximation procedure is used to approximate the model’s expected sufficient statistics.<ref>{{cite web
|url=http://icml2008.cs.helsinki.fi/papers/638.pdf
Line 96 ⟶ 101:
* Teaching hospital press release: {{cite news |title=New AI technology integrates multiple data types to predict cancer outcomes |url=https://medicalxpress.com/news/2022-08-ai-technology-multiple-cancer-outcomes.html |access-date=18 September 2022 |work=[[Brigham and Women's Hospital]] via medicalxpress.com |language=en |archive-date=20 September 2022 |archive-url=https://web.archive.org/web/20220920172825/https://medicalxpress.com/news/2022-08-ai-technology-multiple-cancer-outcomes.html |url-status=live }}</ref>
▲==Multimodal transformers==
▲{{excerpt|Transformer (machine learning model)|Multimodality}}
==See also==
*[[Hopfield network]]
| |||