Google Research的工程师Gaurav Menghani最近发表的综述论文“Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better”。

摘要

Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

深度学习彻底改变了计算机视觉、自然语言理解、语音识别、信息检索等领域。然而，随着深度学习模型的逐步改进，它们的参数数量、延迟、训练所需的资源等都显着增加。因此，关注模型的这些足迹指标也变得很重要，而不仅仅是其质量。我们提出并激发了深度学习中的效率问题，然后对模型效率的五个核心领域（跨越建模技术、基础设施和硬件）及其开创性工作进行了全面调查。我们还提供了一个基于实验的指南和代码，供从业者优化他们的模型训练和部署。我们相信这是高效深度学习领域的第一次全面调查，涵盖从建模技术到硬件支持的模型效率领域。我们希望这份调查能够为读者提供思维模型和对该领域的必要理解，以应用通用效率技术立即获得显着改进，并为他们提供进一步研究和实验的想法，以实现额外的收益。

另外可以参考作者的相关演讲稿：https://drive.google.com/file/d/1H8C9r3G8KUfmtQjW5oHBkytFWqT1rrye/view。

内容中包含的图片若涉及版权问题，请及时与我们联系删除