UNREAL: Unifying Retrieval and Long-Context with a Single Model

AK's Picks LLM Long Context NLP RAG NRES RecSys / IR DRSA
93
2026年10月06日
长上下文推理与检索增强生成（RAG）在证据选择的规模上差异巨大——前者通常仅依赖单个超长提示，后者则需面对整个语料库。我们提出一个问题：是否存在一种统一的、内置于模型内部的机制，能够跨越这一尺度鸿沟，实现从语料库级检索到长上下文推理的通用证据选择？为此，我们提出了“用单一模型统合检索与长上下文”（UNREAL）框架——一种原生于大语言模型（LLM）的证据选择方法，旨在无缝覆盖语料库检索与长上下文推理两大任务。UNREAL 直接利用冻结的底层大语言模型（LLM）内部表征来编码文本块，并从中生成检索查询；其引入的可训练参数不足 50 万，且完全不改动原有模型主干结构。在包含 30 亿词元、2100 万个文本块的维基百科索引上，全部四种基于密集检索和混合检索的 UNREAL 主干模型均显著超越当前最先进的“检索器–重排序器”系统：在 HotpotQA 上，召回率从 49.1% 提升至 73.2%；在 2WikiMultiHopQA 上，从 31.7% 提升至 60.1%；在 MuSiQue 上，亦从 8.8% 提升至 14.4%。当应用于长上下文任务时，同一套证据选择机制可在生成前主动剔除干扰项：在 NoLiMa 基准测试中，于最大上下文长度 128K 词元下，准确率由 1.0% 大幅提升至 24.83%；在 LV-Eval 基准测试中，F1 分数于 256K 词元上下文下由 49.97% 提升至 54.66%。此外，相较于完整上下文推理，UNREAL 自约 32K 词元起即开始降低计算量（FLOPs）与首字输出延迟（time-to-first-token），且随着上下文长度持续增长，性能增益愈发显著。综上所述，这些结果共同确立了“模型内部证据选择”作为语料库检索与证据稀疏型长上下文推理的共通基础。
Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. We ask whether a single model-internal mechanism can select evidence across this range. We introduce UNifying REtrieval And Long-Context with a Single Model (UNREAL), a model-native evidence selection framework to span corpus retrieval and long-context inference. UNREAL encodes chunks and derives retrieval queries directly from the frozen LLM's internal representations. It adds fewer than 500K trainable parameters and leaves the backbone unchanged. On a 3B-token, 21M-chunk Wikipedia index, all four dense and hybrid UNREAL backbones outperform state-of-the-art retriever-reranker systems. The best model raises recall from 49.1% to 73.2% on HotpotQA, from 31.7% to 60.1% on 2WikiMultiHopQA, and from 8.8% to 14.4% on MuSiQue. Applied to long-context tasks, the same selection mechanism removes distractors before generation, raising NoLiMa accuracy from 1.0% to 24.83% at its maximum context length of 128K tokens, and LV-Eval's F1 score from 49.97% to 54.66% at 256K. UNREAL also reduces FLOPs and time-to-first-token relative to full-context inference from roughly 32K tokens onward, with larger gains as context grows. Together, these results establish model-internal evidence selection as a common foundation for corpus retrieval and evidence-sparse long-context inference.
许愿