Universal approximation theorem: Difference between revisions

Revision as of 20:50, 9 January 2025

Revision as of 16:59, 15 January 2025

Line 28:

The bounded depth and bounded width case was first studied by Maiorov and Pinkus in 1999.<ref name="maiorov">{{Cite journal|last1=Maiorov|first1=Vitaly|last2=Pinkus|first2=Allan|date=April 1999|title=Lower bounds for approximation by MLP neural networks|journal=Neurocomputing|volume=25|issue=1–3|pages=81–91|doi=10.1016/S0925-2312(98)00111-8}}</ref> They showed that there exists an analytic sigmoidal activation function such that two hidden layer neural networks with bounded number of units in hidden layers are universal approximators.

In 2018, Guliyev and Ismailov<ref name="guliyev1">{{Cite journal |last1=Guliyev |first1=Namig |last2=Ismailov |first2=Vugar |date=November 2018 |title=Approximation capability of two hidden layer feedforward neural networks with fixed weights |journal=Neurocomputing |volume=316 |pages=262–269 |arxiv=2101.09181 |doi=10.1016/j.neucom.2018.07.075 |s2cid=52285996}}</ref> constructed a smooth sigmoidal activation function providing universal approximation property for two hidden layer feedforward neural networks with less units in hidden layers. In 2018, they also constructed<ref name="guliyev2">{{Cite journal|last1=Guliyev|first1=Namig|last2=Ismailov|first2=Vugar|date=February 2018|title=On the approximation by single hidden layer feedforward neural networks with fixed weights|journal=Neural Networks|volume=98| pages=296–304|doi=10.1016/j.neunet.2017.12.007|pmid=29301110 |arxiv=1708.06219 |s2cid=4932839 }}</ref> single hidden layer networks with bounded width that are still universal approximators for univariate functions. However, this does not apply for multivariable functions.

In 2022, Shen ''et al.''<ref name="guliyev2">{{Citecite journal |last1=GuliyevShen |first1=NamigZuowei |last2=IsmailovYang |first2=VugarHaizhao |last3=Zhang |first3=Shijun |date=FebruaryJanuary 2022 2018|title=On theOptimal approximation byrate singleof hiddenReLU layernetworks feedforwardin neuralterms networksof withwidth fixedand depth weights|journal=NeuralJournal de Mathématiques Pures et Appliquées Networks|volume=98|157 |pages=296–304101–135 |arxiv=2103.00502 |doi=10.1016/j.neunetmatpur.20172021.1207.007|pmid=29301110 |arxiv=1708.06219009 |s2cid=4932839 232075797}}</ref> constructedobtained singleprecise hiddenquantitative layerinformation networkson the withdepth boundedand width thatrequired areto stillapproximate universala approximatorstarget forfunction univariateby functions.deep However,and thiswide doesReLU notneural apply for multivariable functionsnetworks.

<ref>{{cite journal |last1=Shen |first1=Zuowei |last2=Yang |first2=Haizhao |last3=Zhang |first3=Shijun |date=January 2022 |title=Optimal approximation rate of ReLU networks in terms of width and depth |journal=Journal de Mathématiques Pures et Appliquées |volume=157 |pages=101–135 |arxiv=2103.00502 |doi=10.1016/j.matpur.2021.07.009 |s2cid=232075797}}</ref> obtained precise quantitative information on the depth and width required to approximate a target function by deep and wide ReLU neural networks.

=== Quantitative bounds ===