$\texttt{WEAVER}$, Better, Faster, Longer: An Effective World Model for Robotic Manipulation

AK's Picks Embodied AI and Robotics World Models RC
世界模型(WMs,即学习得到的仿真器)对机器人学的潜在影响极为深远——它能在仅需极少真实世界交互的前提下,实现策略评估、策略优化以及测试时规划。为释放这些下游能力,一个世界模型必须同时满足三项核心要求:(i)保真性(即所生成的仿真轨迹须与真实世界行为高度相关);(ii)一致性(即所生成的仿真轨迹在长时间步长下仍保持逻辑连贯与动态稳定);(iii)高效性(即能够快速生成仿真轨迹)。为此,我们提出名为 $\texttt{WEAVER}$(面向具身推理的跨视角世界建模,World Estimation Across Views for Embodied Reasoning)的世界模型架构——该架构首次同步达成上述全部三项要求,并在机器人操作任务上取得当前最优性能。$\texttt{WEAVER}$ 是一种多视角世界模型,通过流匹配(flow-matching)损失函数联合预测未来隐状态(latents)与奖励值(reward values)。我们系统提炼并验证了支撑长时程动态操作任务所需的关键设计要素,涵盖模型架构、记忆机制与预测目标三大方面——而此类任务恰恰是此前各类世界建模方法长期难以攻克的难点。我们在真实机器人硬件平台上部署并验证了 $\texttt{WEAVER}$ 的实际效果:在策略评估任务中,其预测成功率与真实世界成功率的相关系数达 $ρ = 0.870$;在策略优化任务中,相较基线机器人基础模型 $π_{0.5}$,真实世界任务成功率进一步提升了 $38\%$;在测试时规划任务中,真实世界成功率提升 $14\%$,同时推理速度较以往世界模型提升 $5$–$10$ 倍。此外,在分布外(out-of-distribution)场景下的评估中,$\texttt{WEAVER}$ 同样展现出优于先前世界模型的泛化性能。代码、预训练模型及演示视频详见:https://arnavkj1995.github.io/WEAVER/ 。
The potential impacts of world models (WMs, i.e., learned simulators) on robotics are far-reaching -- policy evaluation, policy improvement, and test-time planning -- all with limited real-world interaction. To unlock these downstream capabilities, a WM needs to jointly satisfy three desiderata: $\textit{(i)}$ fidelity (i.e., producing simulated trajectories that correlate with reality), $\textit{(ii)}$ consistency (i.e., producing simulated trajectories that are coherent over long horizons), and $\textit{(iii)}$ efficiency (i.e., producing simulated trajectories quickly). We propose $\texttt{WEAVER}$ (World Estimation Across Views for Embodied Reasoning): a WM architecture that simultaneously achieves all three desiderata, providing state-of-the-art results on robotic manipulation tasks. $\texttt{WEAVER}$ is a multi-view WM trained to predict future latents and reward values via a flow-matching loss. We distill the key design decisions across model architecture, memory, and prediction objectives required to unlock the kinds of long-horizon dynamic manipulation tasks that have confounded prior world modeling approaches. We apply $\texttt{WEAVER}$ in robotic hardware, demonstrating its effectiveness at policy evaluation ($ρ$=0.870 correlation with real-world success rate), policy improvement (real-world success rate improvement of $38\%$ on top of the $π_{0.5}$ robot foundation model), and test-time planning (real-world success rate improvement of $14\%$ with a $5-10\times$ speedup over prior WMs). $\texttt{WEAVER}$ also demonstrates better performance than prior WMs when evaluated on out-of-distribution scenarios. Code, models, and videos at: https://arnavkj1995.github.io/WEAVER/ .
许愿