Content deleted Content added
No edit summary |
|||
Line 6:
Most differentiable programming frameworks work by constructing a graph containing the control flow and [[data structures]] in the program.<ref name="flux">{{cite arXiv |eprint=1811.01457 |last1=Innes |first1=Michael |last2=Saba |first2=Elliot |last3=Fischer |first3=Keno |last4=Gandhi |first4=Dhairya |author5=Marco Concetto Rudilosso |author6=Neethu Mariya Joy |last7=Karmali |first7=Tejan |last8=Pal |first8=Avik |last9=Shah |first9=Viral |title=Fashionable Modelling with Flux |date=2018 |class=cs.PL }}</ref> Attempts generally fall into two groups:
* ''' Static, [[compiled]] graph'''-based approaches such as [[TensorFlow]],<ref group=note>TensorFlow 1 uses the static graph approach, whereas TensorFlow 2 uses the dynamic graph approach by default.</ref> [[Theano (software)|Theano]], and [[MXNet]]. They tend to allow for good [[compiler optimization]] and easier scaling to large systems, but their static nature limits interactivity and the types of programs that can be created easily (e.g. those involving [[loop (computing)|loops]] or [[recursion]]), as well as making it harder for users to reason effectively about their programs.<ref name="flux" /> A proof-of-concept compiler toolchain called Myia uses a subset of Python as a front end and supports higher-order functions, recursion, and higher-order derivatives.<ref>{{cite book |last1=Merriënboer |first1=Bart van |last2=Breuleux |first2=Olivier |last3=Bergeron |first3=Arnaud |last4=Lamblin |first4=Pascal |chapter=Automatic differentiation in ML: where we are and where we should be going |title={{harvnb|NIPS'18}} |date=3 December 2018 |volume=31 |pages=8771–8781 |chapter-url = https://papers.nips.cc/paper/2018/hash/770f8e448d07586afbf77bb59f698587-Abstract.html}}</ref><ref name="myia1">{{Cite web |last1=Breuleux |first1=O. |last2=van Merriënboer |first2=B. |date=2017 |url=https://www.sysml.cc/doc/2018/39.pdf |title=Automatic Differentiation in Myia |access-date=2019-06-24 |archive-date=2019-06-24 |archive-url=https://web.archive.org/web/20190624180156/https://www.sysml.cc/doc/2018/39.pdf |url-status=dead }}</ref><ref name="pytorchtut">{{Cite web|url=https://pytorch.org/tutorials/beginner/examples_autograd/tf_two_layer_net.html |title=TensorFlow: Static Graphs |work=Tutorials: Learning PyTorch |publisher=PyTorch.org |access-date=2019-03-04}}</ref>
* '''[[Operator overloading]], dynamic graph'''-based approaches such as [[PyTorch]], [[NumPy]]'s [[autograd]] package, and [https://darioizzo.github.io/audi/ Pyaudi]. Their dynamic and interactive nature lets most programs be written and reasoned about more easily. However, they lead to [[interpreter (computing)|interpreter]] overhead (particularly when composing many small operations), poorer scalability, and reduced benefit from compiler optimization.<ref name="myia1" /><ref name="pytorchtut" />
The use of just-in-time compilation has emerged recently{{when
A limitation of earlier approaches is that they are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs. Newer approaches resolve this issue by constructing the graph from the language's syntax or IR, allowing arbitrary code to be differentiated.<ref name="flux" /><ref name="myia1" />
==Applications==
Differentiable programming has been applied in areas such as combining [[deep learning]] with [[physics engines]] in [[robotics]],<ref>{{cite arXiv |eprint=1611.01652 |last1=Degrave |first1=Jonas |last2=Hermans |first2=Michiel |last3=Dambre |first3=Joni |last4=wyffels |first4=Francis |title=A Differentiable Physics Engine for Deep Learning in Robotics |date=2016 |class=cs.NE }}</ref> solving [[Quantum chemistry#Electronic structure|electronic-structure]] problems with differentiable [[density functional theory]],<ref name="Li2021">{{cite journal |last1=Li |first1=Li |last2=Hoyer |first2=Stephan |last3=Pederson |first3=Ryan |last4=Sun |first4=Ruoxi |last5=Cubuk |first5=Ekin D. |last6=Riley |first6=Patrick |last7=Burke |first7=Kieron |year=2021 |title=Kohn-Sham Equations as Regularizer: Building Prior Knowledge into Machine-Learned Physics |journal=Physical Review Letters |volume=126 |issue=3 |pages=036401 |arxiv=2009.08551 |bibcode=2021PhRvL.126c6401L |doi=10.1103/PhysRevLett.126.036401 |pmid=33543980 |doi-access=free}}</ref> differentiable [[Ray tracing (graphics)|ray tracing]],<ref>{{cite journal |first1=Tzu-Mao |last1=Li |first2=Miika |last2=Aittala |first3=Frédo |last3=Durand |first4=Jaakko |last4=Lehtinen |title=Differentiable Monte Carlo Ray Tracing through Edge Sampling |journal=ACM Transactions on Graphics |volume=37 |issue=6 |pages=222:1–11 |date=2018 |doi=10.1145/3272127.3275109 |s2cid=52839714 |url=https://people.csail.mit.edu/tzumao/diffrt/|doi-access=free }}</ref> [[differentiable imaging]],<ref>{{Cite journal |last=Chen |first=Ni |last2=Cao |first2=Liangcai |last3=Poon |first3=Ting‐Chung |last4=Lee |first4=Byoungho |last5=Lam |first5=Edmund Y. |title=Differentiable Imaging: A New Tool for Computational Optical Imaging |url=https://onlinelibrary.wiley.com/doi/10.1002/apxr.202200118 |journal=Advanced Physics Research |language=en |volume=2 |issue=6 |doi=10.1002/apxr.202200118 |issn=2751-1200}}</ref>
==Multidisciplinary application==
| |||