作者：S Iwase, S Saito, T Simon, S Lombardi, T Bagautdinov, R Joshi, F Prada, T Shiratori, Y Sheikh, J Saragih
[Meta AI & CMU]

要点:

  1. 提出第一个用于个性化手部模型的神经重照明方法，可以在新的照明条件下进行实时动画；
  2. 采用师生框架，在任意光照下合成高保真的手，进行大量计算，并对自然光照下的外观进行有效的实时预测；
  3. 使用物理学启发的照明特征，如可见度和漫反射阴影，作为神经再照明网络的调节数据，与光传输效应密切相关。

总结:
提出第一个用于高保真个性化手部模型的神经重照明方法，可以在新的照明条件下进行实时动画。

https://arxiv.org/abs/2302.04866

We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage, allowing us to synthesize hands in arbitrary illuminations but with heavy compute. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time. To achieve generalization, we condition the student model with physics-inspired illumination features such as visibility, diffuse shading, and specular reflections computed on a coarse proxy geometry, maintaining a small computational overhead. Our key insight is that these features have strong correlation with subsequent global light transport effects, which proves sufficient as conditioning data for the neural relighting network. Moreover, in contrast to bottleneck illumination conditioning, these features are spatially aligned based on underlying geometry, leading to better generalization to unseen illuminations and poses. In our experiments, we demonstrate the efficacy of our illumination feature representations, outperforming baseline approaches. We also show that our approach can photorealistically relight two interacting hands at real-time speeds.

内容中包含的图片若涉及版权问题，请及时与我们联系删除