Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

LLM Other LLM NLP Benchmarks
2026年10月07日
当前向大语言模型提出的许多问题本身并无唯一正确答案可供评分：例如某项政策的实际价值、用户究竟应选择哪一选项、以及如何权衡相互冲突的价值观等。陈述性偏好经济学（stated-preference economics）数十年来一直面对这一难题。该领域在无法获知真实价值的前提下，依托一套关于“效度”（validity）及相关概念的理论框架来评估调查问卷的应答质量，这些概念包括内容效度（content validity）、结构效度（construct validity）、准则效度（criterion validity）、信度（reliability）、激励相容性（incentive compatibility）以及后果性（consequentiality）。我们认为，这一框架可作为一种通用方法，用于系统评估大语言模型；本文进一步阐明上述每一项概念在大语言模型评估语境下的具体内涵与操作含义。我们以沃斯勒等人（Vossler et al., 2023）发表的一项关于水质改善的陈述性偏好经济价值评估调查为实证案例，将该问卷施测于六种不同大语言模型，从而演示该评估方法的实际应用。在此项经济学应用场景中，效度检验体现为一系列源自经济理论的可验证预测：需求曲线应呈向下倾斜形态；支付意愿（willingness to pay）应随公共品受益范围（scope of the good）及受访者收入水平的变化而发生相应变动。这些检验结果清晰地区分出各模型的表现差异：两款较早发布的模型在家庭年收入为7.5万美元的情境下，甚至未能通过最基础的效度检验；而两款最新发布的模型则成功通过了我们所能设计并评分的所有理论效度检验，但在聚合效度（convergent validity）上却呈现出明显分歧。需要强调的是，通过效度检验仅表明模型输出的答案具有内在逻辑一致性（coherence），并不意味着其答案本身在客观现实中就是“正确”的。
Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
许愿