Some remarks on the connection between generative diffusion and optimal transport

GenAI Diffusion ML Generative Models DM
我们探讨了生成式扩散模型与最优传输之间的三个简单联系。 首先,在任意时刻,数据与其奥恩斯坦–乌伦贝克(Ornstein–Uhlenbeck)加噪过程的联合分布,恰好是某一熵正则化最优传输问题的解;该问题的“温度”参数由当前噪声强度决定。因此,反向扩散过程可被理解为一种退火过程,其作用是逐步降低熵正则化的强度。 其次,我们研究“物种分化”相变现象——即数据分布的各个模态(modes)在生成过程中如何逐步显现——并从最优传输的视角,对比分析扩散模型与流匹配(flow matching)在该过程中的异同。以高维高斯混合模型为例,我们指出:扩散模型与流匹配通过动力学失稳机制实现模态分离,而这一机制在最优传输框架下则转化为生成初期一个指数级狭窄的边界层;该边界层经熵正则化作用而变得平滑、圆润。 第三,针对具有一般约束势能的通用前向动力学过程,我们证明:由得分函数(score)驱动的反向过程同样满足一个基于漂移项的变分原理,且该变分原理的最小代价恰好等于初始分布与最终分布之间的熵差。
We discuss three simple connections between generative diffusion and optimal transport. First, the joint law of the data and of their Ornstein-Uhlenbeck noising is, at every time, the solution of an entropic optimal-transport problem, at a temperature set by the noise level. Backward diffusion can consequently be read as an annealing which gradually decreases the entropic regularization. Second, we study the speciation transition--how the modes of the data distribution emerge during generation--and compare how it happens in diffusion and flow matching with respect to optimal transport. Using Gaussian mixtures as high-dimensional example, we show that the dynamical instability through which diffusion and flow matching separate the modes becomes, for optimal transport, an exponentially thin boundary layer at the beginning of generation, which is rounded by entropic regularization. Third, considering a generic forward dynamics with a general confining potential, we show that the backward process driven by the score also satisfies a drift-based variational principle whose minimal cost is the entropy difference between initial and final distributions.
许愿