Extreme Video Compression with Pre-trained Diffusion Models

2024年02月14日
扩散模型在生成高质量图像和视频数据方面取得了显著的成功。最近,它们也被用于高感知质量的图像压缩。本文提出了一种新颖的极端视频压缩方法,利用解码器中基于扩散的生成模型的预测能力。条件扩散模型采用多个神经压缩帧并生成随后的帧。当重建质量低于所需水平时,新帧被编码以重新启动预测。整个视频按顺序编码以实现视觉上令人愉悦的重建,考虑到感知质量度量,如学习的感知图像补丁相似度(LPIPS)和Frechet视频距离(FVD),比特率低至0.02比特/像素(bpp)。实验结果表明,与标准编解码器(如H.264和H.265)相比,所提出的方案在低bpp范围内的有效性。结果展示了利用生成模型利用视频数据中的时间关系的潜力。代码可在以下网址获得:https://github.com/ElesionKyrie/Extreme-Video-Compression-With-Prediction-Using-Pre-trainded-Diffusion-Models-。
Diffusion models have achieved remarkable success in generating high quality image and video data. More recently, they have also been used for image compression with high perceptual quality. In this paper, we present a novel approach to extreme video compression leveraging the predictive power of diffusion-based generative models at the decoder. The conditional diffusion model takes several neural compressed frames and generates subsequent frames. When the reconstruction quality drops below the desired level, new frames are encoded to restart prediction. The entire video is sequentially encoded to achieve a visually pleasing reconstruction, considering perceptual quality metrics such as the learned perceptual image patch similarity (LPIPS) and the Frechet video distance (FVD), at bit rates as low as 0.02 bits per pixel (bpp). Experimental results demonstrate the effectiveness of the proposed scheme compared to standard codecs such as H.264 and H.265 in the low bpp regime. The results showcase the potential of exploiting the temporal relations in video data using generative models. Code is available at: https://github.com/ElesionKyrie/Extreme-Video-Compression-With-Prediction-Using-Pre-trainded-Diffusion-Models-
许愿