LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

Andrea Yaoyun Cui ,

Hanyu Wang ,

Xinyi Wang ,

Bangzheng Li ,

Lin Zhang ,

Zhenggang Tang ,

Shyamal Buch ,

Dejia Xu ,

Matthew Tancik

Embodied AI and Robotics MLMRD Agent Agent Eval Benchmarks

机器人策略需具备跨部署环境的泛化能力，这包括机器人本体自身的各种变化：摄像头可能被意外碰撞后重新安装、标定参数会随时间发生漂移、控制器运行频率可能发生变化、末端执行器（夹爪）的接触垫也会逐渐磨损。我们提出了 LIBERO-RIG 基准测试集，该基准仅对机器人的硬件配置与系统设置施加扰动，而保持所有 LIBERO 任务本身完全不变。该基准覆盖十一个扰动维度，包括摄像头安装方式与光学特性、摄像头与机械臂之间的标定关系、执行器增益（actuation gain）、控制频率（control rate）、本体感知噪声（proprioceptive noise）、机器人基座位姿、桌面高度以及夹爪接触垫状态。每个扰动维度均设置五个严重程度等级，由此在 LIBERO 的四个任务套件中共构成 210 种不同测试条件。我们采用每种条件 500 轮次（episodes）的规模，对九种近期提出的视觉–语言–动作（vision-language-action）模型进行了系统评估，并通过严格的统计检验对每一项发现予以验证和支撑。整体成功率在中等扰动强度下从 0.932 下降至 0.767，在最高扰动强度下进一步降至 0.509。不同扰动维度及动作接口之间的性能退化程度差异显著；此外，仅在训练阶段引入视觉扰动，并不能缓解那些不改变摄像头图像内容（即图像外观不变）的硬件或系统层面故障。我们已公开发布该基准测试集，并同步提供各测试轮次所对应固定扰动参数的详细清单（manifests）。

Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for each episode.

许愿