SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation

Embodied AI and Robotics RP MLMRD
2026年10月08日
当前前沿的多模态基础模型(例如 GPT-6 Astra)近期展现出直接驱动机器人控制的强大潜力,但在精细操作任务上的性能仍显不足。我们认为,其失效的一个关键原因未必在于策略能力本身不足,而更可能源于空间可观测性(spatial observability)的欠缺——即现有物理相机配置难以充分呈现任务所依赖的关键空间关系。为此,我们提出 SpatialHarness:一种面向测试阶段的具身化辅助框架,它无需对策略进行微调,也无需更改物理感知硬件配置,即可在测试阶段为精细机器人操作提供实时的空间支撑。SpatialHarness 维护一个与真实世界执行过程实时同步的在线仿真场景,自动识别任务所需的关键空间关系,并渲染出能清晰展现这些关系的补充性虚拟视角,从而将相关信息馈送给冻结(frozen)的多模态策略模型。为确保仿真场景在交互过程中持续与真实世界保持对齐,我们设计了“交互感知的场景同步机制”(interaction-aware scene synchronization),该机制可动态区分静态、持握与过渡三种状态模式。我们在四个真实机器人操作任务上对 SpatialHarness 进行了评估,涵盖精确几何对准、物体相对定位以及铰接式物体交互等典型精细操作场景。实验中,我们固定使用同一冻结版 GPT-6 Astra 策略模型,结果表明 SpatialHarness 显著提升了各项任务的成功率:例如,在插拔任务中成功率从 26.7% 提升至 66.7%,在汉诺塔任务中则从 0% 大幅跃升至 100%。上述结果表明,仅通过在测试阶段提升空间可观测性,即可有效释放当前高性能多模态基础模型中已具备但尚未被充分激发的精细操作能力。项目网站:https://emilia113.github.io/SpatialHarness/
Frontier multimodal foundation models (e.g., GPT-6 Astra) have recently shown strong potential for direct robotic control, yet their performance on fine manipulation remains limited. We argue that an important source of failure is not necessarily insufficient policy capability, but insufficient spatial observability, where task-critical spatial relationships may be poorly revealed by the existing physical camera setup. We introduce SpatialHarness, a test-time embodied harness that provides test-time spatial scaffolding for fine robotic manipulation without policy fine-tuning or changes to the physical sensing setup. SpatialHarness maintains an online simulated scene synchronized with real-world execution, identifies task-critical spatial relationships, and renders complementary virtual views that expose them to a frozen multimodal policy. To keep the simulated scene aligned during interaction, we develop interaction-aware scene synchronization that distinguishes static, held, and transition modes. We evaluate SpatialHarness on four real-robot manipulation tasks spanning precise geometric alignment, object-relative placement, and articulated-object interaction. Using the same frozen GPT-6 Astra policy, SpatialHarness substantially improves task success, including from 26.7% to 66.7% on plug insertion and from 0% to 100% on Tower of Hanoi. These results indicate that improving spatial observability at test time can unlock fine-manipulation capabilities already present in strong multimodal foundation models. Project website: https://emilia113.github.io/SpatialHarness/.
许愿