SpectralGPT: Spectral Foundation Model

最近，基础模型因其自监督学习的潜力而受到广泛关注，可能彻底改变视觉表示学习领域。虽然大多数基础模型都是为了有效处理RGB图像而量身定制的，用于各种视觉任务，但是在针对光谱数据的研究中存在明显的差距，光谱数据在场景理解方面尤其在遥感应用中提供了有价值的信息。为了填补这一空白，我们首次创建了一种通用的遥感基础模型，名为SpectralGPT，专门用于使用新颖的3D生成预训练变换器（GPT）处理光谱遥感图像。与现有的基础模型相比，SpectralGPT：1）以渐进式训练方式适应具有不同大小、分辨率、时间序列和区域的输入图像，使得可以充分利用广泛的光谱遥感大数据；2）利用3D令牌生成进行空间-光谱耦合；3）通过多目标重建捕捉光谱顺序模式；4）训练了一百万个光谱遥感图像，产生具有超过6亿个参数的模型。我们的评估强调了预训练的SpectralGPT模型在四个下游任务中取得了显着的性能提升，表明在地球科学领域推进光谱遥感大数据应用的巨大潜力。这四个任务是：单/多标签场景分类、语义分割和变化检测。

The paper aims to fill the gap in research focused on spectral data in the field of visual representation learning by creating a universal RS foundation model named SpectralGPT that can handle spectral RS images for scene understanding in remote sensing applications.

The key idea of the paper is to create a novel 3D generative pretrained transformer (GPT) called SpectralGPT that can accommodate input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, leveraging 3D token generation for spatial-spectral coupling, capturing spectrally sequential patterns via multi-target reconstruction, and training on one million spectral RS images, yielding models with over 600 million parameters.

The paper highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection. The experiments were designed to evaluate the performance of SpectralGPT models on various downstream tasks using different spectral RS datasets. The paper also provides an in-depth analysis of the learned representations and their interpretability. The authors have made their code and pre-trained models publicly available.

Recent related work in this field includes 'Self-Supervised Learning for Hyperspectral Image Classification via Contrastive Attention Mechanism' and 'Spectral-Spatial Feature Learning with Knowledge Distillation for Hyperspectral Image Classification'.