TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

LLM Other LLM Agent Agent Eval Benchmarks
2026年10月05日
我们提出了 TasteVal——一个用于评估前沿模型“科研品位”（research taste）的基准测试。我们将“科研品位”定义为：识别值得研究的科学问题、设计严谨实验、并合理解读实验结果的能力。TasteVal 专门衡量科研品位中的实验能力部分；在给定固定研究问题的前提下，它评估模型能否迭代式地自主设计实验，并基于实验结果得出有效结论。我们将实验层面的科研品位操作化为“计算效率”：若某位“研究者”（Researcher）仅需专家人类一半的串行实验计算量（serial experimental compute），即可达到与后者相同的得分，则其科研品位即为后者的两倍。因此，“实验品位”实质上是实验计算资源的放大系数（multiplier），从而成为预测人工智能发展进程的一项关键输入指标。 TasteVal 包含 8 项全新、高难度、开放式的任务，全面覆盖当前前沿人工智能研发（AI R&D）中的典型挑战。为将“科研品位”与编程能力解耦，我们在评测中将待测模型设定为“研究者”角色，由其迭代式地提出实验方案；而另一名固定不变的“编码员”（Coder）代理则负责具体实现这些实验，并向“研究者”反馈实验结果。“研究者”持续运行，直至耗尽以下任一预算上限：40 个 H100 GPU 小时，或 120 个真实挂钟小时（wall-clock hour）。 我们招募了 24 位人类专家参与评测，每个任务至少有 2 位专家独立完成；对每项任务，均以其中表现最优的一次尝试作为人类专家基线（expert baseline）。我们共评测了 2023 至 2026 年间发布的 20 款模型。其中表现最佳的模型 Opus 5.5 超越了人类专家基线，其计算效率倍率（compute multiplier）达 2.3 倍（95% 置信区间为 1.15–4.37），且单次运行成本仅为人类基线平均成本的约 1/30。 自 2025 年 12 月以来，前沿模型在 TasteVal 上的计算效率倍率大约每 3.0 个月翻一番（95% 置信区间为 1.7–5.0 个月），相较 2023 年至 2025 年 12 月期间的约每 14 个月翻一番，增速显著加快。若改用最终归一化性能（final normalized performance）这一指标来衡量，则前沿模型未出现趋势性跃变，仍保持约每 14.6 个月翻一番的稳定增长节奏。 为确保 TasteVal 的纯净性与评估效度，我们不对外公开任何测试任务。
We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. We operationalize experimental research taste as compute efficiency; a Researcher who reaches the same score as an expert human using half the serial experimental compute has twice the experimental taste. Experimental taste thus acts as a multiplier on experimental compute, making it a key input to forecasts of AI progress. TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D. To isolate taste from coding ability, the model under evaluation acts as a Researcher that iteratively designs experiments while a fixed Coder agent implements them and reports their results. The Researcher executes until either the 40 H100 hour or 120 wall-clock hour budgets are exhausted. We recruit 24 human experts, at least 2 per task, and take the best expert attempt per task as the expert baseline. We evaluate 20 models released between 2023 and 2026. The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost. On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7-5.0), up from every 14 months between 2023 and December 2025. Measured by final normalized performance, frontier models show no trend break, doubling every 14.6 months. To keep TasteVal uncontaminated, we do not release the tasks.
许愿