PAPER

Inductive Bias and Spectral Properties of Single-Head Attention in High Dimensions

ML REG Transformer
我们研究了在由近期提出的注意力索引模型所定义的合成高维序列任务上,训练单头绑定注意力层时的经验风险最小化问题。利用随机矩阵理论、自旋玻璃物理以及近似消息传递方法,我们推导出训练误差和测试误差的精确渐近表达式,确定了插值阈值与恢复阈值,并刻画了所学得权重矩阵的极限谱分布。权重衰减会隐式地引入核范数正则化,从而倾向于得到低秩的查询矩阵和键矩阵。基于这一发现,我们将查询与键矩阵的标准分解式训练方式,与直接对它们的乘积进行逐元素参数化的训练方式进行比较,揭示了分解形式所带来的归纳偏置。值得注意的是,我们预测的谱分布特征重现了大规模Transformer模型中观察到的经验趋势,为这些现象提供了与之相符的理论视角。
We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks, given by the recently introduced attention-indexed model. Using tools from random matrix theory, spin-glass physics, and approximate message passing, we derive sharp asymptotics for training and test errors, locate interpolation and recovery thresholds, and characterize the limiting spectral distribution of the learned weights. Weight decay induces an implicit nuclear-norm regularization, favoring low-rank query and key matrices. Leveraging this, we compare the standard factorized training of query and key matrices with a direct parameterization in which their product is trained element-wise, revealing the inductive bias introduced by the factorized form. Remarkably, the predicted spectral distribution echoes empirical trends reported in large-scale transformers, offering a theoretical perspective consistent with these phenomena.
许愿