Content deleted Content added
→Radial activation functions: rm weasel phrasing "extremely efficient" |
periodic and quadratic |
||
Line 44:
* [[Polyharmonic spline]]s
where <math>\mathbf c</math> is the vector representing the function ''center'' and <math>a</math> and <math>\sigma</math> are parameters affecting the spread of the radius.
=== Other examples ===
Periodic functions can serve as activation functions. Usually the [[Sine wave|sinusoid]] is used, as any periodic function is decomposable into sinusoids by the [[Fourier transform]].<ref>{{Cite journal |last=Sitzmann |first=Vincent |last2=Martel |first2=Julien |last3=Bergman |first3=Alexander |last4=Lindell |first4=David |last5=Wetzstein |first5=Gordon |date=2020 |title=Implicit Neural Representations with Periodic Activation Functions |url=https://proceedings.neurips.cc/paper/2020/hash/53c04118df112c13a8c34b38343b9c10-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=33 |pages=7462–7473}}</ref>
Quadratic activation maps <math>x \mapsto x^2</math>.<ref>{{Citation |last=Flake |first=Gary William |title=Square Unit Augmented Radially Extended Multilayer Perceptrons |date=1998 |work=Neural Networks: Tricks of the Trade |pages=145–163 |editor-last=Orr |editor-first=Genevieve B. |url=https://link.springer.com/chapter/10.1007/3-540-49430-8_8 |access-date=2024-10-05 |place=Berlin, Heidelberg |publisher=Springer |language=en |doi=10.1007/3-540-49430-8_8 |isbn=978-3-540-49430-0 |editor2-last=Müller |editor2-first=Klaus-Robert}}</ref><ref>{{Cite journal |last=Du |first=Simon |last2=Lee |first2=Jason |date=2018-07-03 |title=On the Power of Over-parametrization in Neural Networks with Quadratic Activation |url=https://proceedings.mlr.press/v80/du18a.html |journal=Proceedings of the 35th International Conference on Machine Learning |language=en |publisher=PMLR |pages=1329–1338}}</ref>
=== Folding activation functions ===
| |||