Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs

Donghyeon Joo ,

Sooraj Puthoor ,

Nuwan Jayasena ,

Bahar Asgari

LLM MQIA MoE AI Systems and Hardware OLPP DPS

大语言模型（LLM）工作负载推动了多分区GPU架构的发展，以实现计算与内存容量的可扩展性；然而，此类架构固有的非一致性内存访问（NUMA）特性以及分区间的通信开销，可能加剧资源争用、削弱数据局部性，从而导致内核执行延迟未达最优。为应对此挑战，我们系统分析了当前主流推理引擎中若干关键LLM内核的实现——涵盖权重投影（weight projection）、混合专家（mixture-of-experts）及多种注意力机制（attention variants）——进而对多分区GPU上的数据访问模式进行了全面刻画。首先，我们提出一种内存访问轨迹分析方法，用以精确提取工作组（workgroup）粒度的数据访问行为及其共享特征；随后，借助周期级仿真器（cycle-level simulator）定量评估上述行为对执行延迟的局部性影响。基于该分析框架，我们将LLM内核的操作数划分为三类跨工作组共享模式：全局共享（global）、部分共享（partial）和私有（private），并进一步指出：针对不同类别，所需的优化策略亦截然不同——从简单的工作组级内存绑定（per-workgroup pinning），到需兼顾子组（subgroup）结构的协同调度（subgroup-aware co-scheduling）。我们的研究发现凸显出：在多分区GPU上，亟需面向数据放置感知（placement-aware）的内核编程范式，以及更智能的硬件架构支持，以协同提升计算任务与数据的局部性。

Large language model (LLM) workloads motivate multi-partition GPUs as a path to scaling compute and memory capacity, but their non-uniform memory access characteristics and inter-partition communication can amplify contention and degrade locality, leading to suboptimal kernel latency. To address this, we analyze performance-critical LLM kernel implementations spanning weight projection, mixture-of-experts, and attention variants of state-of-the-art serving engines to present a characterization of data access patterns in multi-partition GPUs. First, we introduce memory trace analysis methodology to derive workgroup-level data access and sharing behavior, then evaluate the locality implications on latency using a cycle-level simulator. Using these tools, we categorize LLM kernel operands into three inter-workgroup sharing patterns (global, partial, or private) and show that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling. Our findings highlight the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.

许愿