PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

NLP RAG Benchmarks Multimodal Intelligence VQA CMAG Agent Agent Memory
将大型语言模型(LLMs)与检索到的网页文本相结合,已成为当前主流范式;然而,网页本身并非天然以纯文本形式存在:现有系统依赖复杂的解析流水线,将HTML线性化处理,并丢弃其原有的布局结构、视觉呈现及格式信息。为此,我们提出PixelRAG——一种全新的检索增强方法,它以网页原本的视觉形态(即像素图像)对其进行表征,并完全在像素空间内完成检索与阅读过程,从而构建端到端架构,彻底规避了文本抽象这一中间环节。据我们所知,PixelRAG是首个以这种原生视觉形式在完整维基百科语料库上运行的处理流程,其数据存储规模达3000万张网页截图,并配备高效的视觉检索索引。该方法基于现有的视觉嵌入模型(即Qwen3-VL-Embedding),并进一步利用精心筛选的对比学习训练数据,在网页截图数据集上对该模型进行微调。检索所得的截图直接作为像素输入送入视觉语言模型(VLM),全程无需任何中间文本转换步骤。实验表明,PixelRAG在各项基准测试中均持续优于“无检索”及“基于文本的RAG”两类基线方法;尤为令人意外的是,即便在NQ和SimpleQA等长期被广泛研究、高度依赖文本的问答任务上,其性能亦显著领先。此外,PixelRAG在多模态开放域问答(如MMSearch)、面向噪声新闻语料库的评测(如LiveVQA)以及智能体式评测(如MoNaCo)中同样表现优异,相较文本基线方法最高可将准确率提升18.1%。最后,像素表征还为RAG引入了一种全新的效率优化维度——图像压缩:在降低图像分辨率的前提下,最多可减少约三分之二的token开销,同时仍能保持原有精度。我们的结果对“网页检索必须依赖文本表征”这一固有认知提出了挑战,表明网页检索增强完全可以直接在其原生视觉形态下运行,并在提升性能的同时兼顾更高效率。
Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting. We introduce PixelRAG, a new retrieval-augmented method that represents websites in their native visual form and performs retrieval and reading entirely in pixel space, enabling an end-to-end architecture that eliminates text abstraction. PixelRAG is, to our knowledge, the first pipeline to operate over a full Wikipedia corpus in this form, scaling to a datastore of 30 million screenshot images with an efficient visual retrieval index. Built on an existing visual embedding model (i.e., Qwen3-VL-Embedding), PixelRAG further fine-tunes this model on screenshot data with carefully curated contrastive training data. Retrieved screenshots are then fed directly as pixel inputs to a VLM, without intermediate text conversion. PixelRAG consistently outperforms both no-retrieval and text-based RAG baselines, most surprisingly on widely studied text-centric tasks such as NQ and SimpleQA. It also achieves strong gains on multimodal open-domain QA (e.g., MMSearch), benchmarks over noisy news corpora (e.g., LiveVQA), and agentic benchmarks (e.g., MoNaCo), improving accuracy by up to 18.1% over text-based baselines. Finally, pixel representations enable a new efficiency lever for RAG through image compression, achieving up to 3x token cost reduction at lower resolutions while maintaining accuracy. Our results challenge the necessity of text representations in web retrieval, suggesting that web RAG can operate directly in the web's native visual form while improving both performance and efficiency.
许愿