Measuring the Gap Between Human and LLM Research Ideas

AK's Picks LLM HDFE Other LLM NLP APGO Benchmarks
663
2026年07月01日
大语言模型（LLMs）正日益被用于启发研究创意，但现有评估方法大多仅针对单个创意，依据其新颖性、可行性或领域专家的偏好进行评判。我们则提出一个不同角度的问题：当前由大语言模型生成的研究创意，与人类研究者实际产出的创意之间究竟存在多大的差距？为刻画这一差距，我们构建了一个面向高质量人类研究论文的大规模创意生成评估框架。对于每一篇论文，我们逆向推断出一小批与其核心创意高度相关的前期文献——这些文献很可能构成了该论文创意的直接灵感来源；随后，我们向大语言模型输入这批论文的标题与摘要，并提示其据此生成一项全新的研究创意。我们提出了一种二维“研究品位”分类体系，从“机会模式”（opportunity pattern）与“研究范式”（research paradigm）两个维度对每一项创意进行刻画，并以此量化人类研究者创意与大语言模型生成创意之间的偏离程度。在对比多种大语言模型所生成的不同创意集合后，我们发现了一种稳定存在的分布性差异：大语言模型生成的创意显著集中于“桥梁型”（bridge-like）机会及“综合式”（synthesis）研究方法；而人类论文所参考的前期文献，则在问题缺口的界定方式与贡献路径的构建方式上展现出更为宽广、更为均衡的分布格局。这一结果表明，尽管能力强大的大语言模型确实能够产出大量合理可行的研究创意，但其创意谱系的整体覆盖范围仍窄于人类研究者的“品位”，且在结构上系统性地偏向特定类型——即更倾向于弥合既有领域间的缝隙，而非探索全新方向或重构基础范式。
LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.
许愿