Content deleted Content added
Journal Tags: Mobile edit Mobile web edit |
→Definition and formulation: codimension is wrong here. |
||
| (36 intermediate revisions by 22 users not shown) | |||
Line 1:
{{Short description|Machine learning framework}}
'''Neural operators''' are a class of [[
The primary application of neural operators is in learning surrogate maps for the solution operators of [[partial differential equation]]s (PDEs),<ref name="NO journal" /><ref name="NO Nature" /> which are critical tools in modeling the natural environment.<ref name="Evans">{{cite book |author-link=Lawrence C. Evans |first=L. C. |last=Evans |title=Partial Differential Equations |publisher=American Mathematical Society |___location=Providence |year=1998 |isbn=0-8218-0772-2 }}</ref><ref>{{cite press release |title=How AI models are transforming weather forecasting: A showcase of data-driven systems |url=https://phys.org/news/2023-09-ai-weather-showcase-data-driven.html |work=phys.org |publisher=European Centre for Medium-Range Weather Forecasts |date=6 September 2023 }}</ref> Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs<ref>{{cite news |last1=Russ |first1=Dan |last2=Abinader |first2=Sacha |title=Microsoft and Accenture partner to tackle methane emissions with AI technology |url=https://azure.microsoft.com/en-us/blog/microsoft-and-accenture-partner-to-tackle-methane-emissions-with-ai-technology/ |work=Microsoft Azure Blog |date=23 August 2023 }}</ref><ref>{{Citation |last1=Li |first1=Zijie |title=Transformer for Partial Differential Equations' Operator Learning |date=2023-04-27 |url=http://arxiv.org/abs/2205.13671 |access-date=2025-06-23 |arxiv=2205.13671 |last2=Meidani |first2=Kazem |last3=Farimani |first3=Amir Barati}}</ref> compared to existing machine learning methodologies while being significantly faster than numerical solvers.<ref name="FNO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Fourier neural operator for parametric partial differential equations |date=2020 |class=cs.LG |eprint=2010.08895 }}</ref><ref>{{cite news |last1=Hao |first1=Karen |title=AI has cracked a key mathematical puzzle for understanding our world |url=https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/ |work=MIT Technology Review |date=30 October 2020 }}</ref><ref>{{cite news |last1=Ananthaswamy |first1=Anil |title=Latest Neural Nets Solve World's Hardest Equations Faster Than Ever Before |url=https://www.quantamagazine.org/latest-neural-nets-solve-worlds-hardest-equations-faster-than-ever-before-20210419/ |work=Quanta Magazine |date=19 April 2021 }}</ref> Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data,<ref>{{cite journal |last1=Sharma |first1=Anuj |last2=Singh |first2=Sukhdeep |last3=Ratna |first3=S. |title=Graph Neural Network Operators: a Review |journal=Multimedia Tools and Applications |date=15 August 2023 |volume=83 |issue=8 |pages=23413–23436 |doi=10.1007/s11042-023-16440-4 }}</ref> and the geosciences.<ref>{{cite journal |last1=Wen |first1=Gege |last2=Li |first2=Zongyi |last3=Azizzadenesheli |first3=Kamyar |last4=Anandkumar |first4=Anima |last5=Benson |first5=Sally M. |title=U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow |journal=Advances in Water Resources |date=May 2022 |volume=163 |pages=104180 |doi=10.1016/j.advwatres.2022.104180 |arxiv=2109.03697 |bibcode=2022AdWR..16304180W }}</ref> In particular, they have been applied to learning stress-strain fields in materials, classifying complex data like spatial transcriptomics, predicting multiphase flow in porous media,<ref>{{cite journal |last1=Choubineh |first1=Abouzar |last2=Chen |first2=Jie |last3=Wood |first3=David A. |last4=Coenen |first4=Frans |last5=Ma |first5=Fei |title=Fourier Neural Operator for Fluid Flow in Small-Shape 2D Simulated Porous Media Dataset |journal=Algorithms |date=2023 |volume=16 |issue=1 |pages=24 |doi=10.3390/a16010024 |doi-access=free }}</ref> and carbon dioxide migration simulations. Finally, the operator learning paradigm allows learning maps between function spaces, and is different from parallel ideas of learning maps from finite-dimensional spaces to function spaces,<ref name="meshfreeflownet">{{cite book |doi=10.1109/SC41405.2020.00013 |chapter=MESHFREEFLOWNET: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework |title=SC20: International Conference for High Performance Computing, Networking, Storage and Analysis |date=2020 |last1=Jiang |first1=Chiyu Lmaxr |last2=Esmaeilzadeh |first2=Soheil |last3=Azizzadenesheli |first3=Kamyar |last4=Kashinath |first4=Karthik |last5=Mustafa |first5=Mustafa |last6=Tchelepi |first6=Hamdi A. |last7=Marcus |first7=Philip |last8=Prabhat |first8=Mr |last9=Anandkumar |first9=Anima |pages=1–15 |isbn=978-1-7281-9998-6 |url=https://resolver.caltech.edu/CaltechAUTHORS:20200526-153937049 }}</ref><ref name="deeponet">{{cite journal |last1=Lu |first1=Lu |last2=Jin |first2=Pengzhan |last3=Pang |first3=Guofei |last4=Zhang |first4=Zhongqiang |last5=Karniadakis |first5=George Em |title=Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators |journal=Nature Machine Intelligence |date=18 March 2021 |volume=3 |issue=3 |pages=218–229 |doi=10.1038/s42256-021-00302-5 |arxiv=1910.03193 }}</ref> and subsumes these settings as special cases when limited to a fixed input resolution.
▲'''Neural operators''' are a class of [[Deep learning|deep learning]] architectures designed to learn maps between infinite-dimensional [[Function space|function spaces]]. Neural operators represent an extension of traditional [[Artificial neural network|artificial neural networks]], marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn [[Operator (mathematics)|operators]] between function spaces; they can receive input functions, and the output function can be evaluated at any discretization.<ref name="NO journal">{{cite journal |last1=Kovachki |first1=Nikola |last2=Li |first2=Zongyi |last3=Liu |first3=Burigede |last4=Azizzadenesheli |first4=Kamyar |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anandkumar |first7=Anima |title=Neural operator: Learning maps between function spaces |journal=Journal of Machine Learning Research |date=2021 |volume=24 |page=1-97 |arxiv=2108.08481 |url=https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf}}</ref>
== Operator learning ==
Understanding and mapping relationships between function spaces has many applications in engineering and the sciences. In particular, [[Abstract differential equation|one can cast the problem]] of solving partial differential equations as identifying a map between function spaces, such as from an initial condition to a time-evolved state. In other PDEs this map takes an input coefficient function and outputs a solution function. Operator learning is a [[
Using traditional machine learning methods, addressing this problem would involve discretizing the infinite-dimensional input and output function spaces into finite-dimensional grids and applying standard learning models, such as neural networks. This approach reduces the operator learning to finite-dimensional function learning and has some limitations, such as generalizing to discretizations beyond the grid used in training.
The primary properties of neural operators that differentiate them from traditional neural networks is discretization invariance and discretization convergence.<ref name="NO journal" />
== Definition and formulation ==
Architecturally, neural operators are similar to feed-forward neural networks in the sense that they are
Neural operators seek to approximate some operator <math>\mathcal{G} : \mathcal{A} \to \mathcal{U}</math> between function spaces <math>\mathcal{A}</math> and <math>\mathcal{U}</math> by building a parametric map <math>\mathcal{G}_\phi : \mathcal{A} \to \mathcal{U}</math>. Such parametric maps <math>\mathcal{G}_\phi</math> can generally be defined in the form
Line 24 ⟶ 19:
<math>\mathcal{G}_\phi := \mathcal{Q} \circ \sigma(W_T + \mathcal{K}_T + b_T) \circ \cdots \circ \sigma(W_1 + \mathcal{K}_1 + b_1) \circ \mathcal{P},</math>
where <math>\mathcal{P}, \mathcal{Q}</math> are the lifting (lifting the codomain of the input function to a higher dimensional space) and projection (projecting the codomain of the intermediate function to the output
<math>(\mathcal{K}_\phi v_t)(x) := \int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy, </math>
Line 30 ⟶ 25:
where the kernel <math>\kappa_\phi</math> is a learnable implicit neural network, parametrized by <math>\phi</math>.
In practice, one is often given the input function to the neural operator at a specific resolution. For instance, consider the setting where one is given the evaluation of <math>v_t</math> at <math>n</math> points <math>\{y_j\}_j^n</math>. Borrowing from [[Nyström method|Nyström integral approximation methods]] such as [[Riemann sum|Riemann sum integration]] and [[
<math>\int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy\approx \sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j}, </math>
Line 38 ⟶ 33:
<math>v_{t+1}(x) \approx \sigma\left(\sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j} + W_t(v_t(y_j)) + b_t(x)\right).</math>
The above approximation, along with parametrizing <math>\kappa_\phi</math> as an implicit neural network, results in the graph neural operator (GNO).<ref name="Graph NO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Neural operator: Graph kernel network for partial differential equations
There have been various parameterizations of neural operators for different applications.<ref name="FNO" /><ref name="Graph NO"
<math>(\mathcal{K}_\phi v_t)(x) = \mathcal{F}^{-1} (R_\phi \cdot (\mathcal{F}v_t))(x), </math>
Line 46 ⟶ 41:
where <math>\mathcal{F}</math> represents the Fourier transform and <math>R_\phi</math> represents the Fourier transform of some periodic function <math>\kappa_\phi</math>. That is, FNO parameterizes the kernel integration directly in Fourier space, using a prescribed number of Fourier modes. When the grid at which the input function is presented is uniform, the Fourier transform can be approximated using the [[Discrete Fourier transform|discrete Fourier transform (DFT)]] with frequencies below some specified threshold. The discrete Fourier transform can be computed using a [[Fast Fourier transform|fast Fourier transform (FFT)]] implementation.
== Training ==
Training neural operators is similar to the training process for a traditional neural network. Neural operators are typically trained in some [[Lp norm]] or [[Sobolev norm]]. In particular, for a dataset <math>\{(a_i, u_i)\}_{i=1}^N</math> of size <math>N</math>, neural operators minimize (a discretization of)
<math>\mathcal{L}_\mathcal{U}(\{(a_i, u_i)\}_{i=1}^N) := \sum_{i=1}^N \|u_i - \mathcal{G}_\theta (a_i) \|_\mathcal{U}^2</math>,
where <math>\|\cdot \|_\mathcal{U}</math> is a norm on the output function space <math>\mathcal{U}</math>. Neural operators can be trained directly using [[
Another training paradigm is associated with physics-informed machine learning. In particular, [[
== See also ==
* [[Neural network (machine learning)|Neural network]]
* [[Physics-informed neural networks]]
* [[Neural field]]
== References ==
{{reflist}}
== External links ==
*[https://github.com/neuraloperator/neuraloperator/ neuralop] – Python library of various neural operator architectures
[[Category:Deep learning]]
| |||