来自今天的爱可可AI前沿推介

[LG] Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features

A S. Chen, Y Lee, A Setlur, S Levine, C Finn
[Stanford University & CMU & UC Berkeley]

Project and Probe: 基于正交特征插值的样本高效域自适应

要点:

  1. Pro² 是一种轻量且样本高效的域自适应方法，它学习一组多样化特征，通过将这些特征与一个小的目标数据集进行插值来自适应目标分布；
  2. Pro² 学习一个线性投影，将预训练的嵌入映射到正交方向，同时对源数据集中的标签进行预测。这一步旨在学习各种预测性特征，以便在分布漂移后至少有一些特征仍然有用；
  3. Pro² 用一个小型目标数据集在这些预测特征的基础上学习一个线性分类器。理论和经验分析表明，Pro² 学习的投影矩阵在信息论意义上是最佳的分类，由于有利的偏差-方差权衡，带来了更好的泛化；
  4. 在四个数据集上的实验表明，与之前的方法(如标准线性探测)相比，在给定有限的目标数据时，Pro² 的性能提高了5-15%。

一句话总结:
提出一种名为 Project and Probe(Pro²)的样本高效域自适应方法，通过用小目标数据集插值这些特征，来学习一组多样化特征并自适应目标分布。Pro² 首先学习线性投影，将预训练的嵌入映射到正交方向，同时预测源数据集中的标签，使用小型目标数据集在这些投影特征上学习线性分类器。理论和实证分析表明，Pro² 在样本效率和泛化方面优于先前的方法。

Conventional approaches to robustness try to learn a model based on causal features. However, identifying maximally robust or causal features may be difficult in some scenarios, and in others, non-causal "shortcut" features may actually be more predictive. We propose a lightweight, sample-efficient approach that learns a diverse set of features and adapts to a target distribution by interpolating these features with a small target dataset. Our approach, Project and Probe (Pro2), first learns a linear projection that maps a pre-trained embedding onto orthogonal directions while being predictive of labels in the source dataset. The goal of this step is to learn a variety of predictive features, so that at least some of them remain useful after distribution shift. Pro2 then learns a linear classifier on top of these projected features using a small target dataset. We theoretically show that Pro2 learns a projection matrix that is optimal for classification in an information-theoretic sense, resulting in better generalization due to a favorable bias-variance tradeoff. Our experiments on four datasets, with multiple distribution shift settings for each, show that Pro2 improves performance by 5-15% when given limited target data compared to prior methods such as standard linear probing.

论文链接：https://arxiv.org/abs/2302.05441
图片
图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除