Neural operators: Difference between revisions

Revision as of 07:03, 30 November 2023

Revision as of 07:05, 30 November 2023

Line 1:

{{Short description|Machine learning framework}}

'''Neural operators''' are a class of [[deep learning]] architectures designed to learn maps between infinite-dimensional [[Function space|function spaces]] <ref name="patel1">{{cite arXiv |last1=Patel |first1=Ravi G. |last2=Desjardins |first2=Olivier |title=Nonlinear integro-differential operator regression with neural networks |date=2018 |class=cs.LG |eprint=1810.08552}}</ref>. Neural operators represent an extension of traditional [[Artificial neural network|artificial neural networks]], marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn [[Operator (mathematics)|operators]] between function spaces; they can receive input functions, and the output function can be evaluated at any discretization.<ref name="NO journal">{{cite journal |last1=Kovachki |first1=Nikola |last2=Li |first2=Zongyi |last3=Liu |first3=Burigede |last4=Azizzadenesheli |first4=Kamyar |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anandkumar |first7=Anima |title=Neural operator: Learning maps between function spaces |journal=Journal of Machine Learning Research |date=2021 |volume=24 |page=1-97 |arxiv=2108.08481 |url=https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf}}</ref>

The primary application of neural operators is in learning surrogate maps for the solution operators of [[Partial differential equation|partial differential equations]] (PDEs),<ref name="NO journal" /> which are critical tools in modeling the natural environment.<ref name="Evans"> {{cite book |author-link=Lawrence C. Evans |first=L. C. |last=Evans |title=Partial Differential Equations |publisher=American Mathematical Society |___location=Providence |year=1998 |isbn=0-8218-0772-2 }}</ref> <ref> X, S. (2023, September 6). How ai models are transforming weather forecasting: A showcase of data-driven systems. Phys.org. https://phys.org/news/2023-09-ai-weather-showcase-data-driven.html </ref> Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs <ref>Kadri Umay, Y. O. (2023, September 20). Microsoft and&nbsp;accenture&nbsp;partner to tackle methane emissions with AI technology. Microsoft Azure Blog. https://azure.microsoft.com/en-us/blog/microsoft-and-accenture-partner-to-tackle-methane-emissions-with-ai-technology/ </ref> compared to existing machine learning methodologies while being significantly faster than numerical solvers.<ref name="patel2">{{cite journal |last1=Patel |first1=Ravi G. |last2=Trask |first2=Nathaniel A. |last3=Wood |first3=Mitchell A. |last4=Cyr |first4=Eric C. |title=A physics-informed operator regression framework for extracting data-driven continuum models |journal=Computer Methods in Applied Mechanics and Engineering |date=January 2021 |volume=373 |pages=113500 |doi=10.1016/j.cma.2020.113500}}</ref><ref name="FNO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Fourier neural operator for parametric partial differential equations |date=2020 |class=cs.LG |eprint=2010.08895 }}</ref><ref>Hao, K. (2021, October 20). Ai has cracked a key mathematical puzzle for understanding our world. MIT Technology Review. https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/ </ref><ref> Ananthaswamy, A., &amp; Quanta Magazine moderates comments to&nbsp;facilitate an informed, substantive. (2021, September 10). Latest neural nets solve world’s hardest equations faster than ever before. Quanta Magazine. https://www.quantamagazine.org/latest-neural-nets-solve-worlds-hardest-equations-faster-than-ever-before-20210419/ </ref> Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data,<ref>Sharma, A., Singh, S. & Ratna, S. Graph Neural Network Operators: a Review. Multimed Tools Appl (2023). https://doi.org/10.1007/s11042-023-16440-4

</ref> and the geosciences.<ref> Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, Sally M. Benson,

U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow,

Line 24:

== Definition and formulation ==

Architecturally, neural operators are similar to feed-forward neural networks in the sense that they are composed of alternating [[Linear map|linear maps]] and non-linearities. Since neural operators act on and output functions, neural operators have been instead formulated as a sequence of alternating linear [[integral operators]] on function spaces and point-wise non-linearities.<ref name="patel1" /><ref name="NO journal" /> Using an analogous architecture to finite-dimensional neural networks, similar [[Universal approximation theorem|universal approximation theorems]] have been proven for neural operators. In particular, it has been shown that neural operators can approximate any continuous operator on a [[Compact space|compact]] set.<ref name="NO journal"/>

Neural operators seek to approximate some operator <math>\mathcal{G} : \mathcal{A} \to \mathcal{U}</math> between function spaces <math>\mathcal{A}</math> and <math>\mathcal{U}</math> by building a parametric map <math>\mathcal{G}_\phi : \mathcal{A} \to \mathcal{U}</math>. Such parametric maps <math>\mathcal{G}_\phi</math> can generally be defined in the form

Line 46:

The above approximation, along with parametrizing <math>\kappa_\phi</math> as an implicit neural network, results in the graph neural operator (GNO).<ref name="Graph NO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Neural operator: Graph kernel network for partial differential equations |date=2020 |class=cs.LG |eprint=2003.03485 }}</ref>

There have been various parameterizations of neural operators for different applications.<ref name="patel2" /><ref name="FNO" /><ref name="Graph NO" /> These typically differ in their parameterization of <math>\kappa</math>. The most popular instantiation is the Fourier neural operator (FNO). FNO takes <math>\kappa_\phi(x, y, v_t(x), v_t(y)) := \kappa_\phi(x-y)</math> and by applying the [[convolution theorem]], arrives at the following parameterization of the kernel integral operator:

<math>(\mathcal{K}_\phi v_t)(x) = \mathcal{F}^{-1} (R_\phi \cdot (\mathcal{F}v_t))(x), </math>

Line 57:

<math>\mathcal{L}_\mathcal{U}(\{(a_i, u_i)\}_{i=1}^N) := \sum_{i=1}^N \|u_i - \mathcal{G}_\theta (a_i) \|_\mathcal{U}^2</math>,

where <math>\|\cdot \|_\mathcal{U}</math> is a norm on the output function space <math>\mathcal{U}</math>. Neural operators can be trained directly using [[backpropagation]] and [[gradient descent]]-based methods <ref name="patel1" />.

Another training paradigm is associated with physics-informed machine learning. In particular, [[physics-informed neural networks]] (PINNs) use complete physics laws to fit neural networks to solutions of PDEs. Extensions of this paradigm to operator learning are broadly called physics-informed neural operators (PINO),<ref name="PINO">{{cite arXiv |last1=Li |first1=Zongyi | last2=Hongkai| first2=Zheng |last3=Kovachki |first3=Nikola | last4=Jin | first4=David | last5=Chen | first5= Haoxuan |last6=Liu |first6=Burigede | last7=Azizzadenesheli |first7=Kamyar |last8=Anima |first8=Anandkumar |title=Physics-Informed Neural Operator for Learning Partial Differential Equations |date=2021 |class=cs.LG |eprint=2111.03794 }}</ref>, where loss functions can include full physics equations or partial physical laws. As opposed to standard PINNs, the PINO paradigm incorporates a data loss (as defined above) in addition to the physics loss <math>\mathcal{L}_{PDE}(a, \mathcal{G}_\theta (a))</math>. The physics loss <math>\mathcal{L}_{PDE}(a, \mathcal{G}_\theta (a))</math> quantifies how much the predicted solution of <math>\mathcal{G}_\theta (a)</math> violates the PDEs equation for the input <math>a</math>.