Neural operators: Difference between revisions

Revision as of 22:49, 18 October 2023

Latest revision as of 14:56, 22 August 2025

(43 intermediate revisions by 24 users not shown)

Line 1:

{{Short description|Machine learning framework}}

{{AfC submission|t||ts=20230821004737|u=Mocl125|ns=118|demo=}}<!-- Important, do not remove this line before article has been created. -->

'''Neural operators''' are a class of [[Deep learning|deep learning]] architecturearchitectures designed to learn maps between infinite-dimensional [[Functionfunction space|function spaces]]s. Neural operators represent an extension of traditional [[Artificialartificial neural network|artificial neural networks]]s, marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn [[Operator (mathematics)|operators]] inbetween function spaces; they can receive input functions, and the output function can be evaluated at any discretization.<ref name="NO journal">{{cite journal |last1=Kovachki |first1=Nikola |last2=Li |first2=Zongyi |last3=Liu |first3=Burigede |last4=Azizzadenesheli |first4=Kamyar |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anandkumar |first7=Anima |title=Neural operator: Learning maps between function spaces |journal=Journal of Machine Learning Research |date=2021 |volume=24 |pagepages=1-971–97 |arxiv=2108.08481 |url=https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf}}</ref><ref name="NO Nature">{{cite journal |last1=Azizzadenesheli |first1=Kamyar |last2=Kovachki |first2=Nikola |last3=Li |first3=Zongyi |last4=Liu-Schiaffini |first4=Miguel |last5=Kossaifi |first5=Jean |last6=Anandkumar |first6=Anima |title=Neural operators for accelerating scientific simulations and design |journal=Nature Reviews Physics |date=2024 |volume=6 |pages=320–328 |arxiv=2309.15325 |url=https://www.nature.com/articles/s42254-024-00712-5}}</ref>

The primary application of neural operators is in learning surrogate maps for the solution operators of [[partial differential equation]]s (PDEs),<ref name="NO journal" /><ref name="NO Nature" /> which are critical tools in modeling the natural environment.<ref name="Evans">{{cite book |author-link=Lawrence C. Evans |first=L. C. |last=Evans |title=Partial Differential Equations |publisher=American Mathematical Society |___location=Providence |year=1998 |isbn=0-8218-0772-2 }}</ref><ref>{{cite press release |title=How AI models are transforming weather forecasting: A showcase of data-driven systems |url=https://phys.org/news/2023-09-ai-weather-showcase-data-driven.html |work=phys.org |publisher=European Centre for Medium-Range Weather Forecasts |date=6 September 2023 }}</ref> Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs<ref>{{cite news |last1=Russ |first1=Dan |last2=Abinader |first2=Sacha |title=Microsoft and Accenture partner to tackle methane emissions with AI technology |url=https://azure.microsoft.com/en-us/blog/microsoft-and-accenture-partner-to-tackle-methane-emissions-with-ai-technology/ |work=Microsoft Azure Blog |date=23 August 2023 }}</ref><ref>{{Citation |last1=Li |first1=Zijie |title=Transformer for Partial Differential Equations' Operator Learning |date=2023-04-27 |url=http://arxiv.org/abs/2205.13671 |access-date=2025-06-23 |arxiv=2205.13671 |last2=Meidani |first2=Kazem |last3=Farimani |first3=Amir Barati}}</ref> compared to existing machine learning methodologies while being significantly faster than numerical solvers.<ref name="FNO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Fourier neural operator for parametric partial differential equations |date=2020 |class=cs.LG |eprint=2010.08895 }}</ref><ref>{{cite news |last1=Hao |first1=Karen |title=AI has cracked a key mathematical puzzle for understanding our world |url=https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/ |work=MIT Technology Review |date=30 October 2020 }}</ref><ref>{{cite news |last1=Ananthaswamy |first1=Anil |title=Latest Neural Nets Solve World's Hardest Equations Faster Than Ever Before |url=https://www.quantamagazine.org/latest-neural-nets-solve-worlds-hardest-equations-faster-than-ever-before-20210419/ |work=Quanta Magazine |date=19 April 2021 }}</ref> Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data,<ref>{{cite journal |last1=Sharma |first1=Anuj |last2=Singh |first2=Sukhdeep |last3=Ratna |first3=S. |title=Graph Neural Network Operators: a Review |journal=Multimedia Tools and Applications |date=15 August 2023 |volume=83 |issue=8 |pages=23413–23436 |doi=10.1007/s11042-023-16440-4 }}</ref> and the geosciences.<ref>{{cite journal |last1=Wen |first1=Gege |last2=Li |first2=Zongyi |last3=Azizzadenesheli |first3=Kamyar |last4=Anandkumar |first4=Anima |last5=Benson |first5=Sally M. |title=U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow |journal=Advances in Water Resources |date=May 2022 |volume=163 |pages=104180 |doi=10.1016/j.advwatres.2022.104180 |arxiv=2109.03697 |bibcode=2022AdWR..16304180W }}</ref> In particular, they have been applied to learning stress-strain fields in materials, classifying complex data like spatial transcriptomics, predicting multiphase flow in porous media,<ref>{{cite journal |last1=Choubineh |first1=Abouzar |last2=Chen |first2=Jie |last3=Wood |first3=David A. |last4=Coenen |first4=Frans |last5=Ma |first5=Fei |title=Fourier Neural Operator for Fluid Flow in Small-Shape 2D Simulated Porous Media Dataset |journal=Algorithms |date=2023 |volume=16 |issue=1 |pages=24 |doi=10.3390/a16010024 |doi-access=free }}</ref> and carbon dioxide migration simulations. Finally, the operator learning paradigm allows learning maps between function spaces, and is different from parallel ideas of learning maps from finite-dimensional spaces to function spaces,<ref name="meshfreeflownet">{{cite book |doi=10.1109/SC41405.2020.00013 |chapter=MESHFREEFLOWNET: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework |title=SC20: International Conference for High Performance Computing, Networking, Storage and Analysis |date=2020 |last1=Jiang |first1=Chiyu Lmaxr |last2=Esmaeilzadeh |first2=Soheil |last3=Azizzadenesheli |first3=Kamyar |last4=Kashinath |first4=Karthik |last5=Mustafa |first5=Mustafa |last6=Tchelepi |first6=Hamdi A. |last7=Marcus |first7=Philip |last8=Prabhat |first8=Mr |last9=Anandkumar |first9=Anima |pages=1–15 |isbn=978-1-7281-9998-6 |url=https://resolver.caltech.edu/CaltechAUTHORS:20200526-153937049 }}</ref><ref name="deeponet">{{cite journal |last1=Lu |first1=Lu |last2=Jin |first2=Pengzhan |last3=Pang |first3=Guofei |last4=Zhang |first4=Zhongqiang |last5=Karniadakis |first5=George Em |title=Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators |journal=Nature Machine Intelligence |date=18 March 2021 |volume=3 |issue=3 |pages=218–229 |doi=10.1038/s42256-021-00302-5 |arxiv=1910.03193 }}</ref> and subsumes these settings as special cases when limited to a fixed input resolution.

'''Neural operators''' are a class of [[Deep learning|deep learning]] architecture designed to learn maps between infinite-dimensional [[Function space|function spaces]]. Neural operators represent an extension of traditional [[Artificial neural network|artificial neural networks]], marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn [[Operator (mathematics)|operators]] in function spaces; they can receive input functions, and the output function can be evaluated at any discretization.<ref name="NO journal">{{cite journal |last1=Kovachki |first1=Nikola |last2=Li |first2=Zongyi |last3=Liu |first3=Burigede |last4=Azizzadenesheli |first4=Kamyar |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anandkumar |first7=Anima |title=Neural operator: Learning maps between function spaces |journal=Journal of Machine Learning Research |volume=24 |page=1-97 |url=https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf}}</ref>

The primary application of neural operators is in learning surrogate maps for the solution operators of [[Partial differential equation|partial differential equations]] (PDEs)<ref name="NO journal" />, which are critical tools in modeling of the natural environment.<ref name="Evans"> {{cite journal |author-link=Lawrence C. Evans |first=L. C. |last=Evans |title=Partial Differential Equations |publisher=American Mathematical Society |___location=Providence |year=1998 |isbn=0-8218-0772-2 }}</ref> Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs compared to existing machine learning methodologies, while being significantly faster than numerical solvers.<ref name="FNO">{{cite journal |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Fourier neural operator for parametric partial differential equations |journal=arXiv preprint arXiv:2010.08895 |date=2020 |url=https://arxiv.org/pdf/2010.08895.pdf}}</ref>

== Operator learning ==

Understanding and mapping relationships between function spaces has many applications in engineering and the sciences. In particular, [[Abstract differential equation|one can cast the problem]] of solving partial differential equations as identifying a map between function spaces, such as from an initial condition to a time-evolved state. In other PDEs this map takes an input coefficient function and outputs a solution function. Operator learning is a [[Machine learning|machine learning]] paradigm to learn solution operators mapping the input function to the output function.

Using traditional machine learning methods, addressing this problem would involve discretizing the infinite-dimensional input and output function spaces into finite-dimensional grids and applying standard learning models, such as neural networks. This approach reduces the operator learning to finite-dimensional function learning and has some limitations, such as generalizing to discretizations beyond the grid used in training.

The primary properties of neural operators that differentiate them from traditional neural networks is discretization invariance and discretization convergence.<ref name="NO journal" />. Unlike conventional neural networks, which are fixed on the discretization of training data, neural operators can adapt to various discretizations without re-training. This property improves the robustness and applicability of neural operators in different scenarios, providing consistent performance across different resolutions and grids.

== Definition and formulation ==

Architecturally, neural operators are similar to feed-forward neural networks in the sense that they are comprisedcomposed of alternating [[Linearlinear map|linear maps]]s and non-linearities. Since neural operators act on and output functions, neural operators have been instead formulated as a sequence of alternating linear [[Integral operators|integral operators]] on function spaces and point-wise non-linearities.<ref name="NO journal" /> Using an analogous architecture to finite-dimensional neural networks, similar [[Universaluniversal approximation theorem|universal approximation theorems]]s have been proven for neural operators. In particular, it has been shown that neural operators can approximate any continuous operator on a [[Compact space|compact]] set.<ref name="NO journal"/>

Neural operators seek to approximate some operator <math>\mathcal{G} : \mathcal{A} \to \mathcal{U}</math> bybetween buildingfunction a parametric mapspaces <math>\mathcal{G}_\theta : \mathcal{A} \to \mathcal{U}</math>. Letand <math>a \in \mathcal{AU}</math>, whereby building a parametric map <math>\mathcal{AG}</math>_\phi denotes: some\mathcal{A} input\to function space. Let <math>\mathcal{U}</math>. denoteSuch theparametric output space and letmaps <math>u \in \mathcal{UG}_\phi</math>. Neural operators arecan generally be defined in the form

<math>\mathcal{G}_\thetaphi := \mathcal{Q} \circ \sigma(W_T + \mathcal{K}_T + b_T) \circ \cdots \circ \sigma(W_1 + \mathcal{K}_1 + b_1) \circ \mathcal{P},</math>

where <math>\mathcal{P}, \mathcal{Q}</math> are the lifting (lifting the codomain of the input function to a higher dimensional space) and projection (projecting the codomain of the intermediate function to the output codimensiondimension) operators, respectively. These operators act pointwise on functions and are typically parametrized as a [[Multilayer perceptron|multilayer perceptron]]s. <math>\sigma</math> is a pointwise nonlinearity, such as a [[Rectifier (neural networks)|rectified linear unit (ReLU)]], or a [[Rectifier (neural networks)#Other_nonOther non-linear_variantslinear variants|Gaussian error linear unit (GeLU)]]. Each layer <math>it=1, \dots, T</math> has a respective local operator <math>W_iW_t</math> (usually parameterized by a pointwise neural network), a kernel integral operator <math>\mathcal{K}_t</math>, and a bias function <math>b_ib_t</math>. Given some intermediate functional representation <math>v_t</math> with ___domain <math>D</math> in athe <math>t</math>-th hidden layer, a kernel integral operator <math>\mathcal{K}_\phi</math> is defined as

<math>(\mathcal{K}_\phi v_t)(x) := \int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy, </math>

where the kernel <math>\kappa_\phi</math> is a learnable implicit neural network, parametrized by <math>\phi</math>.

In practice, one is often given the input function to the neural operator at a specific resolution. For instance for the <math>i</math>th data sample, consider the setting where one is given the evaluation of <math>v_t</math> at <math>n</math> points <math>\{y_j\}_j^n</math>. Borrowing from [[Nyström method|Nyström integral approximation methods]] such as [[Riemann sum|Riemann sum integration]] and [[Gaussian quadrature|Gaussian quadrature]], the above integral operation can be computed as follows:

<math>\int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy\approx \sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j}, </math>

where <math>\Delta_{y_j}</math> is the sub-area volume or quadrature weight associated to the point <math>y_j</math>. Thus, a simplified layer can be computed as

<math>v_{t+1}(x) \approx \sigma\left(\sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j} + W_t(v_t(y_j)) + b_t(x)\right).</math>

The above approximation, along with deployment of implicit neural network forparametrizing <math>\kappa_\phi</math> as an implicit neural network, results in the graph neural operator (GNO).<ref name="Graph NO">{{cite journalarXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Neural operator: Graph kernel network for partial differential equations |journaldate=arXiv preprint arXiv:2003.034852020 |dateclass=2020cs.LG |urleprint=https://arxiv.org/pdf/2003.03485.pdf }}</ref>.

There have been various parameterizations of neural operators for different applications.<ref name="FNO" /><ref name="Graph NO">{{cite journal |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Neural operator: Graph kernel network for partial differential equations |journal=arXiv preprint arXiv:2003.03485 |date=2020 |url=https://arxiv.org/pdf/2003.03485.pdf}}</ref>. These typically differ in their parameterization of <math>\kappa</math>. The most popular instantiation is the Fourier neural operator (FNO). FNO takes <math>\kappa_\phi(x, y, av_t(x), av_t(y))v_t(y) := \kappa_\phi(x-y)</math> and by applying the [[Convolution theorem|convolution theorem]], arrives at the following parameterization of the kernel integrationintegral operator:

<math>(\mathcal{K}_\phi(a) v_t)(x) = \mathcal{F}^{-1} (R_\phi \cdot (\mathcal{F}v_t))(x), </math>

where <math>\mathcal{F}</math> represents the Fourier transform and <math>R_\phi</math> represents the Fourier transform of some periodic function <math>\kappakappa_\phi</math>. That is, FNO parameterizes the kernel integration directly in Fourier space, using a handfulprescribed number of Fourier modes. When the grid at which the input function is presented is uniform, the Fourier transform can be approximated using summation, resulting inthe [[Discrete Fourier transform|discrete Fourier transform (DFT)]] with frequencies atbelow some specified threshold. The discrete Fourier transform can be computed using a [[Fast Fourier transform|fast Fourier transform (FFT)]] implementation.

== Training ==

Training neural operators is similar to the training process for a traditional neural network. Neural operators are typically trained in some [[Lp norm]] or [[Sobolev norm]]. In particular, for a dataset <math>\{(a_i, u_i)\}_{i=1}^N</math> of size <math>N</math>, neural operators minimize (a discretization of)

<math>\mathcal{L}_\mathcal{U}(\{(a_i, u_i)\}_{i=1}^N) := \sum_{i=1}^N \|u_i - \mathcal{G}_\theta (a_i) \|_\mathcal{U}^2</math>,

in some normwhere <math>\|\cdot \|_\mathcal{U}.</math> is a norm on the output function space <math>\mathcal{U}</math>. Neural operators can be trained directly using [[Backpropagation|backpropagation]] and [[Gradient descent|gradient descent]]-based methods.

Another training paradigm is associated with physics-informed machine learning. In particular, [[Physics-informed neural networks|physics-informed neural networks]] (PINNs) use complete physics laws to fit neural networks to solutions of PDEs. The extensionExtensions of this paradigm to operator learning are broadly called physics -informed neural operators (PINO),<ref name="PINO">{{cite journalarXiv |last1=Li |first1=Zongyi | last2=Hongkai| first2=Zheng |last3=Kovachki |first3=Nikola | last4=Jin | first4=David | last5=Chen | first5= Haoxuan |last6=Liu |first6=Burigede | last7=Azizzadenesheli |first7=Kamyar |last8=Anima |first8=Anandkumar |title=Physics-Informed Neural Operator for Learning Partial Differential Equations |journaldate=arXiv preprint arXiv:2111.037942021 |dateclass=2021cs.LG |urleprint=https://arxiv.org/abs/2111.03794 }}</ref>, where loss functions can can include full physics equations or partial physical laws. As opposed to standard PINNs, the PINO paradigm incorporates a data loss (as defined above) in addition to the physics loss <math>\mathcal{L}_{PDE}(a, \mathcal{G}_\theta (a))</math>. The physics loss <math>\mathcal{L}_{PDE}(a, \mathcal{G}_\theta (a))</math> quantifies how much the predicted solution of <math>\mathcal{G}_\theta (a)</math> violates the PDEs equation for the input <math>a</math>.

== See also ==

* [[Neural network (machine learning)|Neural network]]

* [[Physics-informed neural networks]]

* [[Neural field]]

== References ==

<!-- Inline citations added to your article will automatically display here. See en.wikipedia.org/wiki/WP:REFB for instructions on how to add citations. -->

{{reflist}}

== External links ==

*[https://github.com/neuraloperator/neuraloperator/ neuralop] – Python library of various neural operator architectures

[[Category:Deep learning]]