Content deleted Content added
Submitting using AfC-submit-wizard |
→Definition and formulation: codimension is wrong here. |
||
| (27 intermediate revisions by 18 users not shown) | |||
Line 1:
{{Short description|Machine learning framework}}
'''Neural operators''' are a class of [[deep learning]] architectures designed to learn maps between infinite-dimensional [[
The primary application of neural operators is in learning surrogate maps for the solution operators of [[partial differential equation]]s (PDEs),<ref name="NO journal" /><ref name="NO Nature" /> which are critical tools in modeling the natural environment.<ref name="Evans">{{cite book |author-link=Lawrence C. Evans |first=L. C. |last=Evans |title=Partial Differential Equations |publisher=American Mathematical Society |___location=Providence |year=1998 |isbn=0-8218-0772-2 }}</ref><ref>{{cite press release |title=How AI models are transforming weather forecasting: A showcase of data-driven systems |url=https://phys.org/news/2023-09-ai-weather-showcase-data-driven.html |work=phys.org |publisher=European Centre for Medium-Range Weather Forecasts |date=6 September 2023 }}</ref> Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs<ref>{{cite news |last1=Russ |first1=Dan |last2=Abinader |first2=Sacha |title=Microsoft and Accenture partner to tackle methane emissions with AI technology |url=https://azure.microsoft.com/en-us/blog/microsoft-and-accenture-partner-to-tackle-methane-emissions-with-ai-technology/ |work=Microsoft Azure Blog |date=23 August 2023 }}</ref><ref>{{Citation |last1=Li |first1=Zijie |title=Transformer for Partial Differential Equations' Operator Learning |date=2023-04-27 |url=http://arxiv.org/abs/2205.13671 |access-date=2025-06-23 |arxiv=2205.13671 |last2=Meidani |first2=Kazem |last3=Farimani |first3=Amir Barati}}</ref> compared to existing machine learning methodologies while being significantly faster than numerical solvers.<ref name="FNO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Fourier neural operator for parametric partial differential equations |date=2020 |class=cs.LG |eprint=2010.08895 }}</ref><ref>{{cite news |last1=Hao |first1=Karen |title=AI has cracked a key mathematical puzzle for understanding our world |url=https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/ |work=MIT Technology Review |date=30 October 2020 }}</ref><ref>{{cite news |last1=Ananthaswamy |first1=Anil |title=Latest Neural Nets Solve World's Hardest Equations Faster Than Ever Before |url=https://www.quantamagazine.org/latest-neural-nets-solve-worlds-hardest-equations-faster-than-ever-before-20210419/ |work=Quanta Magazine |date=19 April 2021 }}</ref> Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data,<ref>{{cite journal |last1=Sharma |first1=Anuj |last2=Singh |first2=Sukhdeep |last3=Ratna |first3=S. |title=Graph Neural Network Operators: a Review |journal=Multimedia Tools and Applications |date=15 August 2023 |volume=83 |issue=8 |pages=23413–23436 |doi=10.1007/s11042-023-16440-4 }}</ref> and the geosciences.<ref>{{cite journal |last1=Wen |first1=Gege |last2=Li |first2=Zongyi |last3=Azizzadenesheli |first3=Kamyar |last4=Anandkumar |first4=Anima |last5=Benson |first5=Sally M. |title=U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow |journal=Advances in Water Resources |date=May 2022 |volume=163 |pages=104180 |doi=10.1016/j.advwatres.2022.104180 |arxiv=2109.03697 |bibcode=2022AdWR..16304180W }}</ref> In particular, they have been applied to learning stress-strain fields in materials, classifying complex data like spatial transcriptomics, predicting multiphase flow in porous media,<ref>{{cite journal |last1=Choubineh |first1=Abouzar |last2=Chen |first2=Jie |last3=Wood |first3=David A. |last4=Coenen |first4=Frans |last5=Ma |first5=Fei |title=Fourier Neural Operator for Fluid Flow in Small-Shape 2D Simulated Porous Media Dataset |journal=Algorithms |date=2023 |volume=16 |issue=1 |pages=24 |doi=10.3390/a16010024 |doi-access=free }}</ref> and carbon dioxide migration simulations. Finally, the operator learning paradigm allows learning maps between function spaces, and is different from parallel ideas of learning maps from finite-dimensional spaces to function spaces,<ref name="meshfreeflownet">{{cite book |doi=10.1109/SC41405.2020.00013 |chapter=MESHFREEFLOWNET: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework |title=SC20: International Conference for High Performance Computing, Networking, Storage and Analysis |date=2020 |last1=Jiang |first1=Chiyu Lmaxr |last2=Esmaeilzadeh |first2=Soheil |last3=Azizzadenesheli |first3=Kamyar |last4=Kashinath |first4=Karthik |last5=Mustafa |first5=Mustafa |last6=Tchelepi |first6=Hamdi A. |last7=Marcus |first7=Philip |last8=Prabhat |first8=Mr |last9=Anandkumar |first9=Anima |pages=1–15 |isbn=978-1-7281-9998-6 |url=https://resolver.caltech.edu/CaltechAUTHORS:20200526-153937049 }}</ref><ref name="deeponet">{{cite journal |last1=Lu |first1=Lu |last2=Jin |first2=Pengzhan |last3=Pang |first3=Guofei |last4=Zhang |first4=Zhongqiang |last5=Karniadakis |first5=George Em |title=Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators |journal=Nature Machine Intelligence |date=18 March 2021 |volume=3 |issue=3 |pages=218–229 |doi=10.1038/s42256-021-00302-5 |arxiv=1910.03193 }}</ref> and subsumes these settings as special cases when limited to a fixed input resolution.
▲'''Neural operators''' are a class of [[deep learning]] architectures designed to learn maps between infinite-dimensional [[Function space|function spaces]]. Neural operators represent an extension of traditional [[Artificial neural network|artificial neural networks]], marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn [[Operator (mathematics)|operators]] between function spaces; they can receive input functions, and the output function can be evaluated at any discretization.<ref name="NO journal">{{cite journal |last1=Kovachki |first1=Nikola |last2=Li |first2=Zongyi |last3=Liu |first3=Burigede |last4=Azizzadenesheli |first4=Kamyar |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anandkumar |first7=Anima |title=Neural operator: Learning maps between function spaces |journal=Journal of Machine Learning Research |date=2021 |volume=24 |page=1-97 |arxiv=2108.08481 |url=https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf}}</ref>
== Operator learning ==
Understanding and mapping relationships between function spaces has many applications in engineering and the sciences. In particular, [[Abstract differential equation|one can cast the problem]] of solving partial differential equations as identifying a map between function spaces, such as from an initial condition to a time-evolved state. In other PDEs this map takes an input coefficient function and outputs a solution function. Operator learning is a [[machine learning]] paradigm to learn solution operators mapping the input function to the output function.
Using traditional machine learning methods, addressing this problem would involve discretizing the infinite-dimensional input and output function spaces into finite-dimensional grids and applying standard learning models, such as neural networks. This approach reduces the operator learning to finite-dimensional function learning and has some limitations, such as generalizing to discretizations beyond the grid used in training.
The primary properties of neural operators that differentiate them from traditional neural networks is discretization invariance and discretization convergence.<ref name="NO journal" />
== Definition and formulation ==
Architecturally, neural operators are similar to feed-forward neural networks in the sense that they are
Neural operators seek to approximate some operator <math>\mathcal{G} : \mathcal{A} \to \mathcal{U}</math> between function spaces <math>\mathcal{A}</math> and <math>\mathcal{U}</math> by building a parametric map <math>\mathcal{G}_\phi : \mathcal{A} \to \mathcal{U}</math>. Such parametric maps <math>\mathcal{G}_\phi</math> can generally be defined in the form
Line 40 ⟶ 19:
<math>\mathcal{G}_\phi := \mathcal{Q} \circ \sigma(W_T + \mathcal{K}_T + b_T) \circ \cdots \circ \sigma(W_1 + \mathcal{K}_1 + b_1) \circ \mathcal{P},</math>
where <math>\mathcal{P}, \mathcal{Q}</math> are the lifting (lifting the codomain of the input function to a higher dimensional space) and projection (projecting the codomain of the intermediate function to the output
<math>(\mathcal{K}_\phi v_t)(x) := \int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy, </math>
Line 54 ⟶ 33:
<math>v_{t+1}(x) \approx \sigma\left(\sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j} + W_t(v_t(y_j)) + b_t(x)\right).</math>
The above approximation, along with parametrizing <math>\kappa_\phi</math> as an implicit neural network, results in the graph neural operator (GNO).<ref name="Graph NO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Neural operator: Graph kernel network for partial differential equations |date=2020 |class=cs.LG |eprint=2003.03485 }}</ref>
There have been various parameterizations of neural operators for different applications.<ref name="FNO" /><ref name="Graph NO" />
<math>(\mathcal{K}_\phi v_t)(x) = \mathcal{F}^{-1} (R_\phi \cdot (\mathcal{F}v_t))(x), </math>
Line 69 ⟶ 48:
where <math>\|\cdot \|_\mathcal{U}</math> is a norm on the output function space <math>\mathcal{U}</math>. Neural operators can be trained directly using [[backpropagation]] and [[gradient descent]]-based methods.
Another training paradigm is associated with physics-informed machine learning. In particular, [[physics-informed neural networks]] (PINNs) use complete physics laws to fit neural networks to solutions of PDEs. Extensions of this paradigm to operator learning are broadly called physics-informed neural operators (PINO),<ref name="PINO">{{cite arXiv |last1=Li |first1=Zongyi | last2=Hongkai| first2=Zheng |last3=Kovachki |first3=Nikola | last4=Jin | first4=David | last5=Chen | first5= Haoxuan |last6=Liu |first6=Burigede | last7=Azizzadenesheli |first7=Kamyar |last8=Anima |first8=Anandkumar |title=Physics-Informed Neural Operator for Learning Partial Differential Equations |date=2021 |class=cs.LG |eprint=2111.03794 }}</ref>
== See also ==
* [[Neural network (machine learning)|Neural network]]
* [[Physics-informed neural networks]]
* [[Neural field]]
== References ==
{{reflist}}
== External links ==
*[https://github.com/neuraloperator/neuraloperator/ neuralop] – Python library of various neural operator architectures
[[Category:Deep learning]]
| |||