Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

AK's Picks Multimodal Intelligence SDFA GenAI Diffusion
2026年06月09日
基于扩散模型的唇形同步方法虽能实现优异的视觉质量与音画对齐效果,但其依赖全序列双向注意力机制及大量去噪步骤,导致实时推理难以落地。本文提出“唇形强制”(Lip Forcing)方法——据我们所知,这是首个面向视频到视频(V2V)唇形同步任务的自回归式扩散建模方法。该方法将一个参数量达140亿(14B)的、以音频为条件的双向视频扩散教师模型,知识蒸馏为若干因果(即自回归)学生模型。在推理阶段,学生模型仅需两步去噪即可生成每一帧视频块,且无需在推理时采用分类器自由引导(CFG),从而真正实现唇形同步的实时化。 针对唇形同步任务专门设计的教师轨迹分析揭示了一种CFG引导下的保真度–同步性权衡关系:不使用CFG的预测结果更倾向于保持参考视频的细节保真度;而采用CFG引导的预测则在去噪轨迹中段区间内更有利于提升唇形同步精度。Lip Forcing 将这一发现转化为三项分析驱动的核心组件:同步窗口去噪匹配蒸馏(Sync-Window DMD)、两步式推理调度策略,以及基于SyncNet的同步性奖励函数。 我们在两个不同规模的学生模型上验证了Lip Forcing的有效性,二者均源自同一14B教师模型。其中,1.3B规模的学生模型在31帧/秒(FPS)下达到实时流式处理能力,速度较同规模双向扩散模型提升17.6倍;而14B规模的学生模型则是目前文献报道中参数量最大的V2V唇形同步扩散模型,其推理速度相较原始教师模型提升39.8倍,同时在参考视频保真度方面保持相当水平。此外,两种规模模型的首帧延迟均低于1毫秒,显著优于所有现有扩散模型基线。
Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.
许愿