NumBench: Diagnosing Counting Failures in Text-to-Image Models

NLP Benchmarks GenAI TDIG
2026年08月28日
文生图(T2I)模型常生成错误数量的目标物体,但现有基准测试规模过小或控制力度不足,难以揭示其根本原因。为此,我们提出 **NumBench**——一个涵盖64万条提示语的新型基准,覆盖1600个物体类别,目标数量范围为1至100。该基准采用正交实验设计,系统性地调节物体组成、空间布局引导及外观条件,同时在各类别与各数量级之间实现均衡采样。此外,我们构建了一个过程模型:模型将用户请求的各个实例视为在图像中有限且可分辨的空间区域上展开竞争。该模型预测,在低密度(即图像中目标总数较少)情况下,实例间会发生近似二次方量级的“碰撞缺失”(collision deficit);同时表明,通过协调一致的物体排布可有效缓解此类缺失。为支持可扩展的自动化评估,我们提出了“置信度加权数值精度得分”(CWNPS),该指标融合三个经校准的目标计数检测器的输出,并对置信度较低的检测结果予以降权处理。我们在五种商用系统、两种开源模型以及两种专用计数方法上开展评测,结果表明:所有方法的性能均随用户请求的目标数量增加而急剧下降;当目标数量超过50时,所有被测方法均表现薄弱。在各项影响因素中,“目标数量区间”对性能的影响最为显著,其次为布局方式与物体组成。“网格化引导”(grid guidance)在各类受控布局中效果最强,这与前述“协调排布可缓解碰撞缺失”的理论预测高度一致;不过,本研究并未证实“碰撞”是导致计数误差的唯一成因。一项包含14,400张图像的人工评估研究验证了自动化评估方法在目标数量达50以内时的可靠性;而针对243条自然语言提示语的泛化性测试结果则进一步表明,相关发现可超越NumBench模板,推广至更广泛的提示表达形式。
Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce \textbf{NumBench}, a benchmark of 640{,}000 prompts spanning 1{,}600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (\cwnps), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14{,}400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.
许愿