Echo-Memory: A Controlled Study of Memory in Action World Models

AK's Picks Embodied AI and Robotics World Models Agent Agent Memory GenAI Diffusion ML Mamba/SSM Continual Learning Replay Methods
我们提出了**回声记忆(Echo-Memory)**,这是一项针对动作驱动型世界模型中记忆机制的受控研究。此类模型可根据首帧图像、文本提示及相机动作序列生成多段视频,但其核心缺陷往往在于记忆能力不足,而非局部图像合成质量不佳:当相机移开再返回时,场景或关键物体可能悄然发生变化。现有记忆设计难以横向比较,因为性能提升常与主干网络架构、训练策略、检索方式及评估方法等多重因素相互纠缠。回声记忆则固定动作到视频的接口,仅系统性地改变历史信息在生成器中的存储与读取方式。在统一的视频扩散主干网络、优化器、相机动作表征、采样器及评估流程下,我们对比了四种记忆机制:原始上下文(raw context)、基于压缩的记忆(compression-based memory)、采用不同读出路径的空间摘要(spatial summaries),以及状态空间递归(state-space recurrence)。该严格对齐的实验矩阵,将原本混杂交织的四个维度——**容量(capacity)**、**压缩性(compression)**、**读出方式(read-out)** 和 **递归结构(recurrence)**——清晰分离。此外,我们还通过三类评测分支来全面评估记忆能力:**重放质量(replay quality)**、**域内循环重返(in-domain loop revisit)** 和 **开放域返回探测(open-domain return probes)**。三类指标结果常常不一致,表明仅靠重放保真度不足以充分衡量模型对世界的记忆能力。由此得出三项关键发现:第一,原始上下文虽结构简单,却是一个极具竞争力的容量基线,在开放域返回任务上的提升远超其在重放指标上的增益;第二,紧凑性并非容量的“免费替代品”:过度激进的空间压缩与混合压缩记忆会丢失支撑返回任务所需的关键视觉证据;第三,在本实验矩阵中,分块式状态空间递归机制展现出最强的开放域返回能力,表明隐式记忆的**结构设计**本身,与其是否被启用同等重要。上述结果为超越孤立重放指标、深入探究动作驱动型世界模型中记忆机制,提供了一套简洁而系统的评估协议。
We present \textbf{Echo-Memory}, a controlled study of memory mechanisms in action-conditioned world models. These models generate multi-segment videos from a first frame, text prompt, and camera-action sequence, but their central failure is often memory rather than local image synthesis: after the camera leaves and returns, the scene or salient object may silently change. Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences. Echo-Memory fixes the action-to-video interface and varies only how history is stored and read by the generator. Under a shared video diffusion backbone, optimizer, camera-action representation, sampler, and evaluation pipeline, we compare raw context, compression-based memory, spatial summaries with different read-out paths, and state-space recurrence. This matched matrix separates four otherwise conflated axes: \emph{capacity}, \emph{compression}, \emph{read-out}, and \emph{recurrence}. We also evaluate memory through a three-branch protocol: replay quality, in-domain loop revisit, and open-domain return probes. The branches routinely disagree, showing that replay fidelity is not a sufficient proxy for remembering a world. Three findings follow. Raw context is a strong capacity baseline and improves open-domain return far more than it improves replay metrics. Compactness is not a free substitute for capacity: aggressive spatial and hybrid-compression memories lose the salient evidence needed for return. Finally, block-wise state-space recurrence is the strongest open-domain return mechanism in our matrix, showing that the structure of implicit memory matters as much as the decision to use it. These results provide a compact protocol for studying memory in action world models beyond isolated replay metrics.
许愿