Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation

Bum Jun Kim ,

Makoto Kawano ,

Yusuke Iwasawa ,

Yutaka Matsuo

LLM PTSAD MQIA ML TAVR NOR

2026年10月13日

现代Transformer模型依赖多种尺度控制机制，包括归一化（normalization）、残差缩放（residual scaling）、注意力温度系数（attention temperatures）、损失加权（loss weighting）以及自适应参数更新（adaptive updates）。为阐明这些机制如何维持模型内部各张量的尺度一致性，我们提出了“神经量纲分析”（Neural Dimensional Analysis, NDA）——一种基于量纲一致性（dimensional consistency）的统一理论框架。我们定义了“神经量纲单位”（neural units），用以明确张量在前向与反向传播中如何按比例缩放，并规定哪些物理量可以相加（正如物理学中通过单位制约束不同物理量能否相加一样）。该框架所导出的规则自然满足残差连接的量纲兼容性要求，对注意力机制和门控机制的输入实现无量纲化处理，支持跨量纲单位的多任务损失函数协同优化，并能将梯度的量纲单位一致地转换为模型参数的量纲单位。我们在实践中通过三方面落实NDA：其一是引入无量纲的“单位违反指数”（Unit Violation Index, UVI），用于精准诊断Transformer内部各模块间的尺度不匹配问题；其二是构建满足量纲一致性的多损失联合优化目标；其三是设计“规范-正交Transformer”（Gauge-Canonical Transformer, GCT），该架构采用归一化的坐标表示，并在残差连接中实施正交化加法操作。在涵盖5100万至10亿参数规模的语言建模实验中，GCT在训练全程展现出卓越的稳定性：其适用的学习率范围横跨十六倍（即最高学习率可达最低学习率的16倍），且在经调优的学习率下，于1.24亿参数规模模型上验证集损失显著优于Pre-LN、LLaMA风格及QK-Norm等主流基线架构。此外，GCT中经过归一化的残差流亦展现出更强的量化鲁棒性：在10亿参数规模下，采用校准后的逐张量int8量化后，GCT验证集损失仅增加0.009纳特（nats），而各基线模型的损失增幅则高达最多5.51纳特。进一步在视觉任务及物理信息神经网络（physics-informed neural networks）中的实验也充分证实，NDA的益处远不止于语言建模领域。上述结果共同确立了“量纲一致性”作为神经网络设计的一项基本原理。

Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle.

许愿