EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

Embodied AI and Robotics Visual Servoing Active Perception ML RL
2026年10月02日
受人类视觉机制启发,我们提出了一种基于主动凝视(active gaze)的框架,仅借助单个双目相机即可实现细粒度的双手协同操作。EyeRobot 2.0 通过旋转两个“眼”视角,物理性地将视线汇聚于场景中的一个三维注视点(3D fixation point),从而实现对目标位置的精确聚焦;随后,系统采用中心凹式(foveal)图像处理方式——即在图像中心区域分配更多视觉标记(visual tokens),使计算资源集中于与任务高度相关的关键视觉特征上。这种“主动视觉凝视”(Active Visual Fixation, AVF)要求在任务执行过程中对凝视行为进行精细协调;为此,我们采用分层策略予以实现:首先训练一个以目标物体为条件的底层凝视伺服策略(gaze servoing policy),再训练一个目标选择器(target selector),该选择器依据任务进展动态生成下一注视目标。两个模块均基于真实世界数据,采用强化学习(RL)进行训练:其中,凝视伺服策略采用密集的几何奖励函数进行训练;而目标选择器则与行为克隆(BC)驱动的夹爪控制策略联合训练,使其能够在执行任务过程中自主发现一系列注视序列——这些序列在模式和节奏上高度近似人类执行同类任务时的自然凝视轨迹。此外,EyeRobot 2.0 进一步利用注视机制,将夹爪位姿信息统一转换至以注视点为原点的相对 SE(3) 坐标系中,从而显著压缩待学习动作分布的空间维度,提升策略学习效率。我们在7项真实世界任务和6项仿真任务中采集了遥操作(teleoperation)数据,并开展了超过1000次真实机器人实验与1800次仿真机器人实验,将 EyeRobot 2.0 与仅使用被动双目视觉(passive stereo)以及同时使用本体(ego)相机加腕部(wrist)相机的基线策略进行了全面对比——所有策略均在相同数据集上训练。对于标准策略而言,移除腕部相机代价高昂:仅依赖被动双目视觉时,真实场景下的任务成功率从52%骤降至27%。而 EyeRobot 2.0 仅凭双目视觉即有效弥合了这一性能鸿沟,在真实环境中相较被动双目策略提升达40%,在仿真环境中亦提升20%;当腕部相机视野清晰时,其性能与“本体+腕部”相机策略基本持平(69% vs. 64%);而在被操纵物体遮挡腕部相机视野的更具挑战性场景下,EyeRobot 2.0 的成功率更高达48%,是“本体+腕部”策略(22%)的两倍有余。
Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)
许愿