The Impact of Backbone Evolution on LLM-Based Relevance Assessments

Chuting Yu ,

Guido Zuccon ,

Teerapong Leelanupab

LLM Other LLM RecSys / IR LMAE

2026年10月07日

大语言模型（LLMs）正迅速演进，新一代模型展现出更强大的能力。这使人普遍认为：在基于LLM的相关性判断任务中，只要提示词（prompt）保持不变，能力更强的模型将与人类判断达成更高的一致性。然而，我们通过考察LLM相关性判别器在模型主干（backbone）持续迭代过程中的行为，对这一认知提出了挑战。在全程固定提示词的前提下，我们针对一系列连续发布的商用模型（Gemini、GPT）和开源权重模型（Qwen、Llama），分别评估了一种典型的单条提示方法（UMBRELA）和一种基于评分细则的方法（EXAM）。总体而言，我们并未发现任何一致性的证据表明：模型版本越新，其相关性判别能力就越强。尤为关键的是，整体性能持平甚至略有提升，并不意味着判别结果具有稳定性——早期版本LLM主干所作出的正确判断，在后续版本中未必能得到保留。我们进一步探究了此类性能退化现象的潜在成因。研究结果警示我们：不应想当然地认为，为某一特定主干版本设计并验证过的判别提示词，在模型更新（即便同属一个模型家族）后仍能维持同等甚至更优的判别效果。

LLMs are evolving rapidly, with newer models offering stronger capabilities. This suggests that in LLM-based relevance judging, more capable models will achieve higher agreement with human judgements under the same prompt. We challenge this understanding by investigating the behavior of LLM-based relevance judges under backbone evolution. Keeping the prompts fixed, we evaluate a representative single-prompt (UMBRELA) and a rubric-based prompt (EXAM) across sequential model versions of commercial (Gemini, GPT) and open-weight (Qwen, Llama) models. Overall, we find no consistent evidence that newer versions lead to better relevance judges. Crucially, similar or improved aggregate performance does not imply judgment stability: correct judgements made by an earlier version of an LLM backbone are not necessarily preserved by later versions. We investigate the potential drivers of these regressions. Our findings caution against the assumption that judging prompts designed and validated for one backbone version will perform equivalently or better when the model is updated, even within the same family.

许愿