ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Hanjing Ye ,

Tianle Zeng ,

Jiazhao Zhang ,

Shaoan Wang ,

Zibo Zhang ,

Weisi Situ ,

Yuchen Zhou ,

Yonggen Ling ,

Hong Zhang

AK's Picks Embodied AI and Robotics SRT RP Multimodal Intelligence Visual Grounding Other MMI CV ODBO

2026年07月22日

具身视觉跟踪（Embodied Visual Tracking, EVT）要求移动智能体仅依靠搭载的前视摄像头，持续跟随由自然语言描述的特定目标。尽管近期提出的视觉-语言-动作（Vision-Language-Action, VLA）策略将目标识别与轨迹规划统一于单一模型框架内，但其链式思维（Chain-of-Thought, CoT）推理过程通常在抽象的空间隐空间中进行，这类隐空间难以监督，且与显式的图像空间检测结果对齐程度较弱。为解决这一问题，我们提出ReferTrack——一种“先指代、再跟踪”的新范式，依托单个前视摄像头实现EVT任务的可靠 grounding（即语义与视觉空间的精准对齐）。本模型首先从一组已索引的边界框（bounding boxes）中选择目标，随后基于该图像空间中 grounded 的决策，解码出后续跟踪路径的关键航点（waypoints）。为在时间维度上保留目标运动线索，ReferTrack维护一个滑动窗口队列，持续缓存历史帧中已被选中的边界框，并通过“时序-视角-边界框指示符”（Temporal-Viewpoint-BBox Indicator, TVBI）标记，将这些边界框的几何特征注入视觉历史序列。此外，我们还构建了一个定制化的Refer-QA数据集，并采用联合训练方式进一步提升目标识别能力。在EVT-Bench基准测试中，ReferTrack在单视角设置下达到当前最优性能：在单目标跟踪、受干扰跟踪和歧义跟踪三个子任务上的成功率分别达89.4%、73.3%和74.1%，其表现不仅媲美、甚至在目标识别难度较高的任务中超越了多个多摄像头基线方法。最后，我们在足式机器人与人形机器人上的真实场景部署实验，充分验证了该方法优异的仿真到现实（sim-to-real）迁移鲁棒性。代码开源地址：https://github.com/MedlarTea/referTrack。

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.

许愿