Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

LLM Long Context HDFE NLP RAG NRES Agent Tool Learning Agent Memory
大型语言模型(LLM)智能体领域的最新进展,已使得复杂自主式工作流成为可能:模型可自主执行信息检索、调用工具,并在大规模语料库上开展推理,从而代表用户完成各类任务。尽管检索增强生成(RAG)技术在智能体搜索系统中的应用日益广泛,现有文献却尚未系统考察检索策略的选择如何与智能体架构及工具调用范式相互作用。一些关键的实际问题——例如工具输出应以何种形式呈现给模型,以及当检索需应对更多无关上下文文本时性能将如何变化——在智能体循环(agent loop)的研究中仍鲜有探索。本文开展了一项实证研究,包含两项实验。实验一在LongMemEval数据集的116个问题子样本上,对比了“grep”式检索与向量检索的效果;实验采用自研智能体框架Chronos,以及各厂商原生命令行接口(CLI)框架(包括Claude Code、Codex和Gemini CLI),分别测试两种工具结果呈现方式:一种是将工具输出以内联(inline)方式直接嵌入提示词中,另一种则是将工具输出存为独立文件,由模型另行读取。实验二则在仅使用grep检索与仅使用向量检索之间进行对比,并逐步混入越来越多的无关对话历史,使每个查询均被置于愈发冗杂的干扰性文本环境中,而真正相关的段落仅占其中一小部分。在Chronos及各厂商CLI框架下,实验一的结果显示,grep检索的整体准确率普遍高于向量检索;但与此同时,最终得分仍高度依赖于所采用的具体框架及工具调用方式——即便底层对话数据完全相同,不同框架与调用风格带来的性能差异依然显著。
Recent advances in Large Language Model (LLM) agents have enabled complex agentic workflows where models autonomously retrieve information, call tools, and reason over large corpora to complete tasks on behalf of users. Despite the growing adoption of retrieval-augmented generation (RAG) in agentic search systems, existing literature lacks a systematic comparison of how retrieval strategy choice interacts with agent architecture and tool-calling paradigm. Important practical dimensions, including how tool outputs are presented to the model and how performance changes when searches must cope with more irrelevant surrounding text, remain under-explored in agent loops. This paper reports an empirical study organized into two experiments. Experiment 1 compares grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI), for both inline tool results and file-based tool results that the model reads separately. Experiment 2 compares grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter. Across Chronos and the provider CLIs, grep generally yields higher accuracy than vector retrieval in our comparisons in experiment 1; at the same time, overall scores still depend strongly on which harness and tool-calling style is used, even when the underlying conversation data are the same.
许愿