Haptic Repurposing with GenAI

混合现实旨在将数字世界和物理世界融合，创造沉浸式的人机交互体验。尽管有了显著的进展，缺乏逼真的触觉反馈常常会打破沉浸式体验，因为这会在视觉和触觉感知之间创造一种脱节。本文介绍了一种名为“基于GenAI的触觉再利用”的创新方法，通过将任何物理物体转化为适应AI生成的虚拟资产的自适应触觉接口，来增强MR交互。利用最先进的生成式AI模型，该系统捕捉物理物体的2D和3D特征，并通过用户指导的提示生成相应的保持原始物体物理形态的虚拟物体。通过基于模型的物体跟踪，系统可以实时动态地将虚拟资产锚定到物理道具上，使物体可以在视觉上变形成任何用户指定的虚拟物体。本文详细介绍了系统的开发，提供了验证其有效性的可用性研究结果，并探讨了其在显著增强交互式MR环境方面的潜力。希望这项工作能为进一步研究沉浸式和触觉技术中的基于AI驱动的空间转换奠定基础。

Haptic feedback is often absent in mixed reality, leading to a disconnect between visual and tactile perceptions. The paper aims to enhance MR interactions by transforming physical objects into adaptive haptic interfaces for AI-generated virtual assets.

The system utilizes generative AI models to capture 2D and 3D features of physical objects and generate corresponding virtual objects that maintain the physical form of the original objects. Through model-based object tracking, virtual assets are anchored to physical props in real time, allowing objects to visually morph into any user-specified virtual object.

Usability studies validate the effectiveness of the system. The paper explores the potential for AI-driven spatial transformation in immersive and haptic technologies. The system has the potential to significantly enhance interactive MR environments.

Related work includes research on haptic feedback in mixed reality, generative AI models, and object tracking.