An introduction to reinforcement learning for neuroscience

强化学习在神经科学中有着丰富的历史，从早期关于多巴胺作为时间差分学习的奖励预测误差信号的研究（Schultz等人，1997）到最近的研究表明，多巴胺可以实现一种在深度学习中流行的“分布式强化学习”（Dabney等人，2020）。在这个领域的文献中，强化学习的理论进展与神经科学实验和发现之间存在着紧密的联系。因此，描述我们实验数据的理论变得越来越复杂和难以理解。在本综述中，我们介绍了经典强化学习理论的基础，并介绍了现代深度强化学习中使用的方法的入门概述，这些方法在系统神经科学中找到了应用。我们首先概述了强化学习问题和经典的时间差分算法，然后讨论了“无模型”和“有模型”的强化学习，以及介于这两类之间的方法，如DYNA和继承表示法。在这些部分中，我们强调了机器学习方法与实验和理论神经科学相关工作之间的密切相似之处。然后，我们介绍了深度强化学习，并举例说明这些方法如何被用来模拟系统神经科学文献中的不同学习现象，如元强化学习（Wang等人，2018）和分布式强化学习（Dabney等人，2020）。本文还提供了实现讨论的方法并生成图表的代码。

The paper aims to provide an introductory overview of methods used in modern deep reinforcement learning that have found applications in systems neuroscience. It also attempts to highlight the close parallels between the machine learning methods and related work in both experimental and theoretical neuroscience.

The key idea of the paper is to provide a comprehensive review of reinforcement learning in neuroscience, covering classical work in reinforcement learning and building up to an introductory overview of methods used in modern deep reinforcement learning that have found applications in systems neuroscience. The paper highlights the close parallels between the machine learning methods and related work in both experimental and theoretical neuroscience.

The paper covers the basic theory underlying classical work in reinforcement learning and provides an introduction to deep reinforcement learning with examples of how these methods have been used to model different learning phenomena in the systems neuroscience literature. The paper also provides code that implements the methods discussed in this work and generates the figures. The paper highlights the close parallels between the machine learning methods and related work in both experimental and theoretical neuroscience.

Recent related work in this field includes 'Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm' by D. Duvenaud et al. (2019), 'Deep reinforcement learning with relational inductive biases' by T. Kipf et al. (2019) and 'A deep neural network model for reinforcement learning in complex environments' by Y. Li et al. (2017).