Sense & Sensitivities: The Path to General-Purpose Algorithmic Differentiation

Part of Proceedings of Machine Learning and Systems 2020 (MLSys 2020)

Bibtex »Metadata »Paper »

Authors

Mike Innes

Abstract

We present Zygote, an algorithmic differentiation (AD) system for the Julia language. Zygote is designed to address the needs of both the machine learning and scientific computing communities, who have historically been siloed by their very different tools. As well as fostering increased collaboration between these communities, we wish to enable \textit{differentiable programming} ($\partial P$), in which arbitrary numerical programs can make use of gradient-based optimisation. We present and evaluate our proposed solutions to the performance/expressiveness tradeoffs in current systems, as well as our work applying AD to many common programming language features, which is applicable to work in other languages and systems.