Round and Round We Go! What makes Rotary Positional Encodings useful?

LLM PTSAD Other LLM
2024年10月08日
位置编码(PEs)是基于Transformer的大型语言模型(LLMs)的关键组成部分,为注意力机制提供重要的序列位置信息。目前LLMs中最流行的编码类型之一是旋转位置编码(RoPE),它根据查询和键的相对距离进行旋转。普遍认为RoPE有用的原因是,随着相对距离的增加,它有助于减少令牌依赖性。在本文中,我们认为这不太可能是核心原因。我们研究了经过训练的Gemma 7B模型的内部,以了解RoPE在机械层面上的使用方式。我们发现Gemma通过利用最高频率来构建稳健的“位置”注意力模式来使用RoPE。我们还发现,总体上,Gemma非常喜欢使用RoPE的最低频率,我们怀疑这些频率用于携带语义信息。我们在数学上证明了RoPE的有趣行为,并进行了实验以验证我们的发现,提出了一种修改RoPE的方法,以解决一些突出问题并提高性能。我们认为,这项工作代表了更好地理解LLMs中PEs的有趣进展,我们认为这对于将LLMs扩展到大尺寸和上下文长度具有关键价值。
Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.
许愿