SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

AK's Picks Embodied AI and Robotics SRT RP RC Sim-Platform
在真实世界中训练和评估机器人策略成本高昂,且难以规模化。我们提出 SimFoundry——一种模块化、全自动的系统,能够仅凭一段视频实现“零样本”的真实场景到仿真场景构建。SimFoundry 可生成即用型仿真数字孪生体,并支持对物体、场景及任务进行灵活编辑,从而实现多样化“数字近亲”(digital cousins)的自动化生成;这些“数字近亲”是原始真实场景在保持功能可供性(affordance)前提下的各类变体。基于 SimFoundry 数据训练所得的策略,可零样本迁移至一系列极具挑战性的现实任务中,涵盖多步骤操作、铰接式物体交互以及双手协同操作等复杂场景;而其生成的“数字近亲”(即原始场景、物体与任务的各类变体)则进一步助力策略泛化至全新的真实世界环境。在涵盖 7 类操作任务与 5 种策略架构的广泛实验中,SimFoundry 的仿真评估结果与真实世界性能高度一致:平均皮尔逊相关系数达 0.911,平均最大排序偏差仅为 0.018。当将纯仿真环境中训练的策略直接零样本部署于真实世界时,若训练阶段分别引入物体级、场景级和任务级的“数字近亲”,其任务成功率相较基线分别平均提升 17%、21% 和 40%。更多技术细节请参见:https://research.nvidia.com/labs/gear/simfoundry/。
Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .
许愿