What Makes a Good Query? Measuring the Impact of Human-Confusing Linguistic Features on LLM Performance

AK's Picks LLM HDFE HDCE NLP APGO
大语言模型(LLM)的幻觉现象通常被视为模型自身或其解码策略存在的缺陷。本文借鉴经典语言学理论指出,查询语句的形式本身亦可影响听者(以及模型)的应答方式。我们据此提出可操作化的研究路径:构建一个包含22个维度的查询特征向量,涵盖从句复杂度、词汇罕见性、回指、否定、可回答性以及意图锚定等各项指标——这些因素在人类语言理解研究中已被证实具有显著影响。基于369,837条真实世界查询数据,我们提出核心问题:是否存在某些特定类型的查询,会系统性地提高模型产生幻觉的概率?大规模实证分析揭示出一幅稳定的“风险图谱”:诸如深层嵌套从句、指代信息不足等特征,与更高的幻觉发生率呈一致正相关;而意图表达清晰、问题本身具备明确可回答性,则与更低的幻觉率显著关联。另一些特征(如领域专指性)则呈现出混合效应,其影响方向和强度因具体数据集与所用模型而异。综上,本研究首次确立了一种可经实证观测的查询特征表征体系,该体系与幻觉风险存在稳健的相关性,从而为面向降低幻觉的查询重写技术提供了理论依据与实践基础,并为后续干预性研究开辟了新路径。
Large Language Model (LLM) hallucinations are usually treated as defects of the model or its decoding strategy. Drawing on classical linguistics, we argue that a query's form can also shape a listener's (and model's) response. We operationalize this insight by constructing a 22-dimension query feature vector covering clause complexity, lexical rarity, and anaphora, negation, answerability, and intention grounding, all known to affect human comprehension. Using 369,837 real-world queries, we ask: Are there certain types of queries that make hallucination more likely? A large-scale analysis reveals a consistent "risk landscape": certain features such as deep clause nesting and underspecification align with higher hallucination propensity. In contrast, clear intention grounding and answerability align with lower hallucination rates. Others, including domain specificity, show mixed, dataset- and model-dependent effects. Thus, these findings establish an empirically observable query-feature representation correlated with hallucination risk, paving the way for guided query rewriting and future intervention studies.
许愿