Differentiable programming: Difference between revisions

Revision as of 12:11, 26 July 2024

Revision as of 12:12, 26 July 2024

Line 9:

* '''[[Operator overloading]], dynamic graph''' based approaches such as [[PyTorch]], [[NumPy]]'s [[autograd]] package as well as [https://darioizzo.github.io/audi/ Pyaudi]. Their dynamic and interactive nature lets most programs be written and reasoned about more easily. However, they lead to [[interpreter (computing)|interpreter]] overhead (particularly when composing many small operations), poorer scalability, and reduced benefit from compiler optimization.<ref name="myia1" /><ref name="pytorchtut" />

The use of Just-in-Time compilation has emerged recently as a possible solution to overcome some of the bottlenecks of interpreted languages. The C++ [https://bluescarni.github.io/heyoka/index.html heyoka] and python package [https://bluescarni.github.io/heyoka.py/index.html heyoka.py] [https://bluescarni.github.io/heyoka/index.html heyoka] make large use of this technique to offer advanced differentiable programming capabilities (also at high orders). A package for the [[Julia (programming language)|Julia]] programming language{{snd}} [https://github.com/FluxML/Zygote.jl Zygote]{{snd}} works directly on Julia's [[intermediate representation]]. <ref name="flux" /><ref>{{cite preprint |arxiv=1810.07951 |last1=Innes |first1=Michael |title=Don't Unroll Adjoint: Differentiating SSA-Form Programs |date=2018 }}</ref><ref name="diffprog-zygote" />

A limitation of earlier approaches is that they are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs. Newer approaches resolve this issue by constructing the graph from the language's syntax or IR, allowing arbitrary code to be differentiated. <ref name="flux" /><ref name="myia1" />