Towards Automating Scientific Review with Google's Paper Assistant Tool

AK's Picks Agent Human Agent Interaction Autonomous Workflows AI Apps / Applied AI EQT
人工智能正推动一场科学发现领域的革命,大幅加速了从假说生成到数学定理证明的整个科研流程。然而,这种迅猛发展也带来了一个系统性挑战:传统依赖人工的同行评审机制,已无法跟上人工智能辅助科研成果激增的步伐。要从根本上化解这一矛盾,我们必须同样借助人工智能来加速科研成果的验证与评审过程本身。为厘清这一转型路径,我们提出了一种关于人工智能与人类在科学评价中协作模式的分类框架,该框架包含四个渐进式层级,并深入探讨了每一层级所涉及的各类权衡取舍。 作为迈向这一未来的重要一步,我们推出了“论文助手工具”(Paper Assistant Tool, PAT)——一个专为深度科学评审与验证而设计的具身型人工智能框架。PAT可完整接收学术论文全文,并输出一份全面、细致的评估报告,内容涵盖理论结果的核查、实验结果的验证、改进建议的提出,以及潜在缺陷的识别。通过采用推理扩展技术(inference scaling techniques),PAT能够比单次模型调用更深入地挖掘问题,在SPOT基准测试中对数学错误的召回率较零样本(zero-shot)方法提升了34%。目前,PAT已在计算机科学领域两大顶级会议——理论计算机年会(STOC)与国际机器学习大会(ICML)——面向作者开展预投稿试点应用,结果表明其具备切实识别关键性错误并提出实质性修改建议的能力。通过及早发现错误,PAT显著减轻了审稿人的认知负荷,同时充分保障审稿人对评审最终结果的决策权与控制力。
Artificial intelligence is driving a revolution in scientific discovery, accelerating everything from hypothesis generation to mathematical theorem proving. However, this rapid acceleration is creating a systemic challenge: traditional human peer review cannot scale to match the influx of AI-assisted science. Ultimately, to resolve this tension, we must also deploy AI to accelerate the verification and review process itself. To frame the discussion around this transition, we propose a taxonomy consisting of four progressive levels of AI-human collaboration in scientific evaluation, and discuss various trade-offs involved with each. As a step toward this future, we introduce the Paper Assistant Tool (PAT), an agentic AI framework built for deep scientific review and verification. PAT ingests full scientific manuscripts and produces a comprehensive evaluation, checking theoretical results, validating experiments, suggesting improvements, and identifying potential flaws. By utilizing inference scaling techniques, PAT is able to identify deeper issues than a single model call alone, achieving a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. Pilot deployments of PAT as a pre-submission tool for authors at two major Computer Science conferences -- STOC and ICML -- demonstrate its ability to identify critical errors and suggest substantive improvements to research papers. By catching errors early, PAT eases the cognitive burden placed on referees, while preserving their control over the outcomes of the review process.
许愿