A Survey on Vision Mamba: Models, Applications and Challenges

ML Mamba/SSM CV ODBO Backbone
2024年04月29日
Mamba是最近一种选择性结构化状态空间模型,对于长序列建模任务表现出色。Mamba通过全局感受野和动态加权,缓解了卷积神经网络的建模限制,并提供了类似于Transformer的先进建模能力。关键是,它在不产生Transformer通常伴随的二次计算复杂度的情况下实现了这一点。由于相对于前两种主流基础模型的优势,Mamba表现出成为视觉基础模型的巨大潜力。研究人员正在积极将Mamba应用于各种计算机视觉任务,导致出现了许多新兴的研究。为了跟上计算机视觉的快速发展,本文旨在提供对视觉Mamba方法的全面回顾。本文首先阐述了原始Mamba模型的公式。随后,我们对视觉Mamba的回顾深入探讨了几个代表性的骨干网络,以阐明视觉Mamba的核心见解。然后,我们使用不同的模态对相关工作进行分类,包括图像、视频、点云、多模态等。具体来说,对于图像应用,我们进一步将它们组织成不同的任务,以促进更有结构的讨论。最后,我们讨论了视觉Mamba的挑战和未来研究方向,为这个快速发展的领域的未来研究提供了见解。本文中回顾的视觉Mamba模型的全面列表可在https://github.com/Ruixxxx/Awesome-Vision-Mamba-Models上找到。
Mamba, a recent selective structured state space model, performs excellently on long sequence modeling tasks. Mamba mitigates the modeling constraints of convolutional neural networks and offers advanced modeling capabilities similar to those of Transformers, through global receptive fields and dynamic weighting. Crucially, it achieves this without incurring the quadratic computational complexity typically associated with Transformers. Due to its advantages over the former two mainstream foundation models, Mamba exhibits great potential to be a visual foundation model. Researchers are actively applying Mamba to various computer vision tasks, leading to numerous emerging works. To help keep pace with the rapid advancements in computer vision, this paper aims to provide a comprehensive review of visual Mamba approaches. This paper begins by delineating the formulation of the original Mamba model. Subsequently, our review of visual Mamba delves into several representative backbone networks to elucidate the core insights of the visual Mamba. We then categorize related works using different modalities, including image, video, point cloud, multi-modal, and others. Specifically, for image applications, we further organize them into distinct tasks to facilitate a more structured discussion. Finally, we discuss the challenges and future research directions for visual Mamba, providing insights for future research in this quickly evolving area. A comprehensive list of visual Mamba models reviewed in this work is available at https://github.com/Ruixxxx/Awesome-Vision-Mamba-Models.
许愿