DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video

我们提出了DINO-Tracker——一种用于视频中长期稠密跟踪的新框架。我们方法的支柱是将单个视频的测试时间训练与预训练的DINO-ViT模型学习到的强大的局部语义特征相结合。具体而言,我们的框架同时采用DINO的特征来适应测试视频的运动观察,同时训练一个直接利用精细特征的跟踪器。整个框架使用自监督损失和正则化的组合进行端到端训练,这使我们能够保留并受益于DINO的语义先验知识。广泛的评估表明,我们的方法在已知基准测试中取得了最先进的结果。DINO-Tracker明显优于自监督方法,并且在长期遮挡跟踪的挑战性情况下表现优于最先进的监督跟踪器,同时具有竞争力。
We present DINO-Tracker -- a new framework for long-term dense tracking in video. The pillar of our approach is combining test-time training on a single video, with the powerful localized semantic features learned by a pre-trained DINO-ViT model. Specifically, our framework simultaneously adopts DINO's features to fit to the motion observations of the test video, while training a tracker that directly leverages the refined features. The entire framework is trained end-to-end using a combination of self-supervised losses, and regularization that allows us to retain and benefit from DINO's semantic prior. Extensive evaluation demonstrates that our method achieves state-of-the-art results on known benchmarks. DINO-tracker significantly outperforms self-supervised methods and is competitive with state-of-the-art supervised trackers, while outperforming them in challenging cases of tracking under long-term occlusions.
许愿