|
'''Neural operators''' are a class of [[deep learning]] architectures designed to learn maps between infinite-dimensional [[Function space|function spaces]]. Neural operators represent an extension of traditional [[Artificial neural network|artificial neural networks]], marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn [[Operator (mathematics)|operators]] between function spaces; they can receive input functions, and the output function can be evaluated at any discretization.<ref name="NO journal">{{cite journal |last1=Kovachki |first1=Nikola |last2=Li |first2=Zongyi |last3=Liu |first3=Burigede |last4=Azizzadenesheli |first4=Kamyar |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anandkumar |first7=Anima |title=Neural operator: Learning maps between function spaces |journal=Journal of Machine Learning Research |date=2021 |volume=24 |page=1-97 |arxiv=2108.08481 |url=https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf}}</ref>
The primary application of neural operators is in learning surrogate maps for the solution operators of [[Partial differential equation|partial differential equations]] (PDEs),<ref name="NO journal" />, which are critical tools in modeling the natural environment.<ref name="Evans"> {{cite book |author-link=Lawrence C. Evans |first=L. C. |last=Evans |title=Partial Differential Equations |publisher=American Mathematical Society |___location=Providence |year=1998 |isbn=0-8218-0772-2 }}</ref> <ref> X, S. (2023, September 6). How ai models are transforming weather forecasting: A showcase of data-driven systems. Phys.org. https://phys.org/news/2023-09-ai-weather-showcase-data-driven.html </ref> Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs <ref>Kadri Umay, Y. O. (2023, September 20). Microsoft and accenture partner to tackle methane emissions with AI technology. Microsoft Azure Blog. https://azure.microsoft.com/en-us/blog/microsoft-and-accenture-partner-to-tackle-methane-emissions-with-ai-technology/ </ref> compared to existing machine learning methodologies while being significantly faster than numerical solvers.<ref name="FNO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Fourier neural operator for parametric partial differential equations |date=2020 |class=cs.LG |eprint=2010.08895 }}</ref><ref>Hao, K. (2021, October 20). Ai has cracked a key mathematical puzzle for understanding our world. MIT Technology Review. https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/ </ref><ref> Ananthaswamy, A., & Quanta Magazine moderates comments to facilitate an informed, substantive. (2021, September 10). Latest neural nets solve world’s hardest equations faster than ever before. Quanta Magazine. https://www.quantamagazine.org/latest-neural-nets-solve-worlds-hardest-equations-faster-than-ever-before-20210419/ </ref>. Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data ,<ref>Sharma, A., Singh, S. & Ratna, S. Graph Neural Network Operators: a Review. Multimed Tools Appl (2023). https://doi.org/10.1007/s11042-023-16440-4
</ref>, and the geosciences.<ref> Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, Sally M. Benson,
U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow,
Advances in Water Resources,
https://doi.org/10.1016/j.advwatres.2022.104180.
(https://www.sciencedirect.com/science/article/pii/S0309170822000562)
</ref> In particular, they have been applied to learning stress-strain fields in materials, classifying complex data like spatial transcriptomics, predicting multiphase flow in porous media,<ref> Choubineh A, Chen J, Wood DA, Coenen F, Ma F. Fourier Neural Operator for Fluid Flow in Small-Shape 2D Simulated Porous Media Dataset. Algorithms. 2023; 16(1):24. https://doi.org/10.3390/a16010024
</ref>, and climate modeling through long-term weather forecasting<ref> Yang, Q., Hernandez-Garcia, A., Harder, P., Ramesh, V., Sattegeri, P., Szwarcman, D., ... & Rolnick, D. (2023). Fourier Neural Operators for Arbitrary Resolution Climate Data Downscaling. arXiv preprint arXiv:2305.14452.</ref> and carbon dioxide migration simulations. Finally, the operator learning paradigm allows learning maps between function spaces, and is different from parallel ideas of learning maps from finite-dimensional spaces to function spaces,<ref name="meshfreeflownet">{{cite journal | vauthors=((Esmaeilzadeh, S., Azizzadenesheli, K., Kashinath, K., Mustafa, M., Tchelepi, H. A., Marcus, P., Prabhat, M., Anandkumar, A., others)) | title=Meshfreeflownet: A physics-constrained deep continuous space-time super-resolution framework | pages=1–15 | publisher=IEEE | date=19 October 2020| arxiv=2005.01463 }}</ref><ref name="deeponet">{{cite journal | vauthors=((Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G. E.)) | title=Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators | volume=3 | issue=3 | pages=218–229 | publisher=Nature Publishing Group UK London | date=19 October 2021}}</ref>, and subsumes these settings when limited to fixed input resolution.
== Operator learning ==
Using traditional machine learning methods, addressing this problem would involve discretizing the infinite-dimensional input and output function spaces into finite-dimensional grids and applying standard learning models, such as neural networks. This approach reduces the operator learning to finite-dimensional function learning and has some limitations, such as generalizing to discretizations beyond the grid used in training.
The primary properties of neural operators that differentiate them from traditional neural networks is discretization invariance and discretization convergence.<ref name="NO journal" />. Unlike conventional neural networks, which are fixed on the discretization of training data, neural operators can adapt to various discretizations without re-training. This property improves the robustness and applicability of neural operators in different scenarios, providing consistent performance across different resolutions and grids.
== Definition and formulation ==
<math>v_{t+1}(x) \approx \sigma\left(\sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j} + W_t(v_t(y_j)) + b_t(x)\right).</math>
The above approximation, along with parametrizing <math>\kappa_\phi</math> as an implicit neural network, results in the graph neural operator (GNO).<ref name="Graph NO">{{cite arXiv |last1=Li |first1=Zongyi |last2=Kovachki |first2=Nikola |last3=Azizzadenesheli |first3=Kamyar |last4=Liu |first4=Burigede |last5=Bhattacharya |first5=Kaushik |last6=Stuart |first6=Andrew |last7=Anima |first7=Anandkumar |title=Neural operator: Graph kernel network for partial differential equations |date=2020 |class=cs.LG |eprint=2003.03485 }}</ref>.
There have been various parameterizations of neural operators for different applications.<ref name="FNO" /><ref name="Graph NO" />. These typically differ in their parameterization of <math>\kappa</math>. The most popular instantiation is the Fourier neural operator (FNO). FNO takes <math>\kappa_\phi(x, y, v_t(x), v_t(y)) := \kappa_\phi(x-y)</math> and by applying the [[convolution theorem]], arrives at the following parameterization of the kernel integral operator:
<math>(\mathcal{K}_\phi v_t)(x) = \mathcal{F}^{-1} (R_\phi \cdot (\mathcal{F}v_t))(x), </math>
|
