Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

AK's Picks Embodied AI and Robotics World Models MLMRD
2026年06月15日
我们提出了Qwen-RobotWorld——一种面向具身智能的语言驱动型视频世界模型。该模型以自然语言作为统一的动作接口,能够基于当前观测,跨机器人操作、自动驾驶、室内导航及人机迁移等多类具身任务,预测具备物理合理性的未来视觉轨迹。这一统一建模框架为三大应用方向提供了坚实基础:一是生成合成数据,用于策略训练的数据增强;二是构建可扩展的虚拟环境,支撑策略性能评估;三是提供语言引导的规划信号,赋能下游机器人控制任务。上述能力通过三部分协同设计实现: a)双流MMDiT架构(融合多模态大语言模型动作编码):采用60层双流扩散Transformer,将冻结的Qwen2.5-VL语义表征与视频变分自编码器(video-VAE)隐空间表征,通过逐层联合注意力机制进行耦合; b)具身世界知识库(Embodied World Knowledge, EWK):包含860万条视频-文本对(逾2亿帧),覆盖20余种具身形态与500余类动作类别,建立了细粒度的动作-语言映射关系; c)通用+专家渐进式课程学习(General+Expert Progressive Curriculum):采用两阶段训练策略,第一阶段学习通用视觉先验知识,第二阶段在共享语言接口下注入具身领域专业知识。 大量实验结果表明,Qwen-RobotWorld展现出卓越竞争力:在EWMBench与DreamGen Bench两项基准上综合排名均位列第一;在WorldModelBench与PBench上全面超越所有开源模型;此外,在RoboTwin-IF基准上的零样本分析进一步验证了其强大的泛化能力与多视角一致性表现。
We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.
许愿