Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

Embodied AI and Robotics SRT MLMRD Agent Agent Planning Autonomous Workflows
在多种任务中构建可靠的机器人执行能力,需要大量人力投入,用于技能开发与维护、奖励函数设计,以及感知与控制模块的协同集成。本文提出“重构—练习—走向真实”(Reconstruct, Practice, Go Real,简称RPG)框架,旨在不更新模型权重的前提下,实现机器人执行系统的自主持续优化。RPG首先从离线数据集中识别出各类操作能力,并据此在仿真环境中构建相应的练习任务;在练习过程中,RPG综合利用执行反馈、仿真器提供的特权状态(privileged simulator state)以及数据集中的视频片段,对失败案例进行归因诊断;基于诊断结果,RPG可生成新的、可复用的符号化技能,优化已有技能,并动态修订系统级提示词(system prompt);所有候选修改均需经过跨任务评估——即分别测试单个修改项及合并后的修订方案的效果,仅当性能提升显著时才予以保留并纳入复用库;在实际部署阶段,一个具备多模态理解能力的大语言模型(multimodal LLM)将依托最终确定的系统提示词与技能库,统筹协调感知模块与机器人底层控制。在22个操作任务的预留初始状态(held-out initializations)上,RPG经15轮练习后,任务成功率由首轮练习后的28.6%大幅提升至95.0%,全面超越所有对比基线方法,包括ASPIRE(75.5%)和搭载GPT-6 Astra Pro大模型的CaP-Agent0(60.0%)。在完成统一校准与硬件适配流程后,该冻结权重(frozen)的系统在全部30次实体机器人实验中均成功完成任务:其中三项任务各执行10次,全部达标。项目网站:https://rpg-robot.github.io/
Building reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model weights. RPG identifies manipulation capabilities in an offline dataset and constructs related practice tasks in simulation. During practice, RPG uses execution feedback, privileged simulator state, and available dataset videos to diagnose failures. It develops new reusable symbolic skills, refines existing skills, and revises the system prompt based on these diagnoses. Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse. At test time, a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. On held-out initializations of 22 manipulation tasks, RPG improves task success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming all evaluated baselines, including ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, with ten trials on each of three tasks. Project Website: https://rpg-robot.github.io/
许愿