ariance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations

G Mialon, R Balestriero, Y LeCun
[Meta AI]

用方差协方差正则化增强自监督表示的成对独立性

要点:

  1. VCReg 是一种自监督学习方法，使 projector 输出协方差矩阵正规化；
  2. 这种正则化在习得表示特征间强制实现了成对独立性；
  3. 成对独立性可使需要线性探测的下游任务的性能得到改善；
  4. VCReg 还可用于独立成分分析。

一句话总结:
方差协方差正则化(VCReg)增强了自监督学习中特征间的成对独立性，进而提高了下游任务的性能。

Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector’s output. This study highlights important properties of such strategy, which we coin Variance-Covariance regularization (VCReg). More precisely, we show that VCReg enforces pairwise independence between the features of the learned representation. This result emerges by bridging VCReg applied on the projector’s output to kernel independence criteria applied on the projector’s input. This provides the first theoretical motivations and explanations of VCReg. We empirically validate our findings where (i) we put in evidence which projector’s characteristics favor pairwise independence, (ii) we use these findings to obtain nontrivial performance gains for VICReg, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis. We hope that our findings will support the adoption of VCReg in SSL and beyond.

论文链接：https://openreview.net/forum?id=Nn-7OXvqmSW
图片

图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除