The Elements of Differentiable Programming

人工智能近来取得了显著进展，这得益于大型模型、海量数据集、加速硬件以及可微编程的变革性能力。这种新的编程范式使得复杂计算机程序（包括具有控制流和数据结构的程序）的端到端微分成为可能，从而实现了基于梯度的程序参数优化。作为一种新兴的编程范式，可微编程基于计算机科学和应用数学的多个领域，包括自动微分、图形模型、优化和统计学。本书全面回顾了可微编程的基本概念。我们采用了两个主要视角，即优化和概率，两者之间有明显的类比关系。可微编程不仅仅是程序的微分，还包括旨在进行微分的程序的深思熟虑的设计。通过使程序可微分，我们本质上引入了其执行的概率分布，提供了一种量化与程序输出相关的不确定性的手段。

Differentiable Programming: The Key to Flexible Machine Learning

Differentiable programming enables end-to-end differentiation of complex computer programs, making gradient-based optimization of program parameters possible. The paper presents a comprehensive review of the fundamental concepts useful for differentiable programming, adopting two main perspectives: optimization and probability.

The paper highlights the importance of thoughtful design of programs intended for differentiation. It also introduces probability distributions over program execution, providing a means to quantify the uncertainty associated with program outputs. The experiments are not discussed in detail, but the paper provides a list of open-source libraries for differentiable programming. The authors suggest that differentiable programming has the potential to revolutionize machine learning and artificial intelligence research.

Related work includes recent advances in automatic differentiation, graphical models, optimization, and statistics. Some related papers include 'Auto-Differentiating Linear Algebra' by Barak Pearlmutter, 'Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks' by Charles Blundell et al., and 'Differentiable Probabilistic Programming for Deep Bayesian Neural Networks' by Jan-Willem van de Meent et al.