How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?

Yuxin Chen ,

Zongyang Ma ,

Ziqi Zhang ,

Zhongang Qi ,

Chunfeng Yuan ,

Bing Li ,

Junfu Pu ,

Ying Shan ,

Xiaojuan Qi ,

Weiming Hu

Multimodal Intelligence Vision-Language Pre-training FGCMA RecSys / IR Semantic Retrieval and Personalization DRSA

主导的双编码模型可以实现高效的图像-文本检索，但准确性有限，而交叉编码器模型则以更高的准确性为代价提供更高的效率。从交叉编码器中提取跨模态匹配知识，用于双编码器，是利用它们的优势的自然方法。因此，我们研究了以下有价值的问题：如何使交叉编码器成为双编码器的良好教师？我们的发现有三个：（1）交叉编码器的跨模态相似度分数分布更集中，而双编码器的结果几乎是正常的，使得香草逻辑蒸馏不太有效。然而，排名蒸馏仍然是实用的，因为它不受分数分布的影响。（2）只有困难样本之间的相对顺序传递有效知识，而易样本之间的顺序信息几乎没有意义。（3）保持蒸馏损失和双编码器训练损失之间的协调有益于知识转移。基于这些发现，我们提出了一种新颖的对比偏序蒸馏（CPRD）方法，它通过对比学习实现了模仿困难负样本之间相对顺序的目标。这种方法与双编码器的训练相协调，有效地将有效的知识从交叉编码器转移到双编码器。对图像-文本检索和排名任务进行的大量实验表明，我们的方法超越了其他蒸馏方法，并显着提高了双编码器的准确性。

Dominant dual-encoder models enable efficient image-text retrieval but suffer from limited accuracy while the cross-encoder models offer higher accuracy at the expense of efficiency. Distilling cross-modality matching knowledge from cross-encoder to dual-encoder provides a natural approach to harness their strengths. Thus we investigate the following valuable question: how to make cross-encoder a good teacher for dual-encoder? Our findings are threefold:(1) Cross-modal similarity score distribution of cross-encoder is more concentrated while the result of dual-encoder is nearly normal making vanilla logit distillation less effective. However ranking distillation remains practical as it is not affected by the score distribution.(2) Only the relative order between hard negatives conveys valid knowledge while the order information between easy negatives has little significance.(3) Maintaining the coordination between distillation loss and dual-encoder training loss is beneficial for knowledge transfer. Based on these findings we propose a novel Contrastive Partial Ranking Distillation (CPRD) method which implements the objective of mimicking relative order between hard negative samples with contrastive learning. This approach coordinates with the training of the dual-encoder effectively transferring valid knowledge from the cross-encoder to the dual-encoder. Extensive experiments on image-text retrieval and ranking tasks show that our method surpasses other distillation methods and significantly improves the accuracy of dual-encoder.

许愿