FIT-RAG: Black-Box RAG with Factual Information and Token Reduction

由于参数数量极大，对大型语言模型（LLMs）进行微调以更新长尾或过时的知识在很多应用中是不可行的。为了避免微调，我们可以将LLM视为黑盒（即冻结LLM的参数），并使用检索增强生成（RAG）系统来增强它，即黑盒RAG。最近，黑盒RAG在知识密集型任务中取得了成功，并引起了广泛关注。现有的黑盒RAG方法通常微调检索器以迎合LLMs的偏好，并将所有检索到的文档连接在一起作为输入，这存在两个问题：（1）忽略事实信息。LLM优选的文档可能不包含所提问的事实信息，这可能会误导检索器并损害黑盒RAG的有效性；（2）浪费标记。简单地连接所有检索到的文档会带来大量不必要的标记，这会降低黑盒RAG的效率。为解决这些问题，本文提出了一种新颖的黑盒RAG框架，它利用了检索中的事实信息，并减少了增强所需的标记数量，称为FIT-RAG。FIT-RAG通过构建一个双标签文档评分器来利用事实信息。此外，它通过引入自我知识识别器和子文档级标记减少器来减少标记。FIT-RAG在三个开放领域的问答数据集（TriviaQA、NQ和PopQA）上进行了广泛的实验验证，实现了优越的有效性和效率。FIT-RAG可以使Llama2-13B-Chat在TriviaQA上的回答准确率提高14.3％，在NQ上提高19.9％，在PopQA上提高27.5％。此外，它可以平均节省约一半的标记。

FIT-RAG: Factual Information and Token Reduction for Black-box Retrieval-Augmented Generation

FIT-RAG proposes a novel black-box RAG framework that utilizes factual information in retrieval and reduces the number of tokens for augmentation, achieving superior effectiveness and efficiency.

The FIT-RAG framework achieves improved answering accuracy and saves approximately half of the tokens on average across three open-domain question-answering datasets. The framework introduces a bi-label document scorer to utilize factual information and a self-knowledge recognizer and sub-document-level token reducer to reduce tokens. The experiments were conducted on TriviaQA, NQ, and PopQA datasets. The paper also discusses the limitations of the proposed framework and suggests future research directions.

Related work includes existing black-box RAG methods that fine-tune retrievers to cater to LLMs' preferences and concatenate all the retrieved documents as input, which suffers from issues of ignorance of factual information and waste of tokens. Other related studies include recent advancements in open-domain question-answering and retrieval-augmented generation.