It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

LLM Other LLM NLP APGO Benchmarks AI Safety / AI Ethics AVAR RA
2026年07月20日
用户经常向大语言模型（LLM）表达其信念。在某些情况下，LLM 应当将这些上下文中的信念视为真实；而在另一些情况下，则应坚持自身既有的先验知识。值得注意的是，用户表达信念的方式（EoBs）在语言形式上具有高度多样性——可能借助预设（presuppositions）、证据标记（evidential markers）与确定性标记（certainty markers），或采用不同语调——而每种表达形式对 LLM 的说服力亦各不相同。为此，我们提出一套类型学框架，用以系统评估各类 EoBs 如何影响模型在“遵循上下文”与“坚守先验知识”之间的权衡取舍。该类型学基于四个具有语言学依据的维度构建：表达形式（form）、证据性（evidentiality）、认知立场（epistemic stance）和语调（tone），共涵盖 17 种精细划分的类型。我们将这些 EoBs 与世界知识事实相配对，生成受控的 EoB–问题对（EoB-query pairs），从而精准分离并考察语言形式差异所引发的独立效应。依托这一评测基准，我们对 16 个 LLM 进行了全面评估，这些模型在架构（Llama3、Qwen3、Gemma3）、规模（参数量从 10 亿至 300 亿不等）以及训练阶段（基础模型 vs 指令微调模型）等方面均存在差异。结果表明，上述各维度均显著影响模型的响应行为：例如，更大规模的模型及经过指令微调的模型，相较于更小规模的模型和基础模型，往往更倾向于依赖自身先验知识，而较少顺应上下文。此外，我们还识别出若干特定类型的 EoBs，其在统计意义上始终比其他类型更能稳定地影响语言模型的判断。本研究揭示了语言表述方式如何系统性地影响 LLM 对上下文信息的整合机制，相关发现对提示工程（prompt engineering）优化及模型鲁棒性提升均具有重要启示意义。
Users frequently express their beliefs to large language models (LLMs). In some situations, the LLM should accept these contextual beliefs as true. In others, they should stick to their prior knowledge. Notably, users' expressions of belief (EoBs) can take linguistically diverse forms - using presuppositions, evidential and certainty markers, or varied tones - each of which may have a different persuasiveness over the LLMs. We introduce a typology to systematically evaluate how different EoBs affect whether models follow context versus prior knowledge. The typology is grounded in four linguistically motivated dimensions: form, evidentiality, epistemic stance, and tone, spanning 17 fine-grained types. By pairing these EoBs with world knowledge facts, we generate controlled EoB-query pairs that isolate the effect of linguistic variation. Using this benchmark, we evaluate 16 LLMs that differ in architecture (Llama3, Qwen3, Gemma3), scale (1B-30B parameters), and training stages (base vs instruct). We identify meaningful variations in response behavior across these axes, e.g., that bigger models and instruction models tend to be less context-following than smaller models and base models. We further identify specific EoBs that statistically significantly persuade LMs more consistently than others. Our work reveals systematic patterns in how linguistic framing affects LLM context integration, with implications for prompt engineering and model robustness.
许愿