Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable Programming

Abstract

Training of deep learning models depends on gradient descent and end-to-end differentiation. Under the slogan of differentiable programming, there is an increasing demand for efficient automatic gradient computation for emerging network architectures that incorporate dynamic control flow, especially in NLP.