Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Embodied AI and Robotics SRT MLMRD CV 3DGS
12
2026年10月09日
用户将一个通用型操作策略带入自家或办公室后，自然期望该策略能从其书架、操作台和桌面上拾取各类物品。然而，即便像 $π_{0.5}$ 这样性能强劲的通用策略，在未经适配的情况下，于我们实际采集的部署场景中拾取成功率仍为 0%；而若要求在每位用户的实际环境中逐一收集真实示范数据，则成本高得令人无法承受。为此，我们提出 Splat2Act——一种全自动的“真实世界→仿真→真实世界”（real-to-sim-to-real）流水线，仅需一部手持智能手机对环境进行扫描，并拍摄用户自有物品的图像，即可将预训练的视觉–语言–动作（VLA）策略个性化适配至特定场景及其对应的一组物体。 与以往仅针对单个桌面场景开展真实→仿真重建的工作不同，Splat2Act 面向的是整栋建筑尺度的室内环境：它利用三维高斯溅射（3D Gaussian Splatting）技术构建具备照片级真实感的数字孪生体，并同步生成可用于碰撞检测的几何模型；自动识别出所有桌面类及非桌面类的可放置区域（即“收纳面”，receptacles）；据此构建切实可行的拾取任务；最后，借助具备完全感知能力的仿真“特权神谕”（privileged simulation oracle）自动生成成功执行这些任务的示范数据——整个流程仅需人类提供场景扫描数据与物体图像，其余全部自动化完成。 我们将该流水线应用于 101 个校园内实际扫描的环境，这些环境的中位可通行面积为 104.87 平方米，平均每处含 7 个收纳面；最终共识别出 852 个收纳面，并生成逾 68,000 条拾取操作示范数据。我们对 $π_{0.5}$ 进行两阶段微调：第一阶段，使其适配数百个公开的物体资产，所用数据或仅来自目标部署场景本身，或整合全部 101 个场景的物体资源；第二阶段，则进一步针对用户自有物体，在其部署场景内的具体收纳面上完成精细化个性化。 最终得到的最强个性化策略，在目标真实场景中的综合拾取成功率达到 62.5%。当该策略部署于真实的 Stretch 3 机器人平台时，在包含 12 项任务的子集上实现了 63.9% 的成功率，与其在仿真环境中针对同一任务集所取得的 66.7% 成功率高度吻合。上述结果表明，仅依靠仿真“神谕”生成的数据，即可通过全自动 real-to-sim-to-real 训练流程，高效实现 VLA 操作策略的个性化定制。
A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $π_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle, with only scene and object capture required from a human. Applied to 101 scanned campus environments with a median navigable area of 104.87 m$^2$ and 7 receptacles, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. We fine-tune $π_{0.5}$ in two stages: Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes; Stage 2 personalizes to the user's own objects on the deployment scene's receptacles. Our strongest personalized policy reaches 62.5% aggregate pickup success in the target scene. Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks. These results suggest that automated real-to-sim-to-real training can personalize VLA manipulation policies using only simulation-oracle data.
许愿