Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication

AI Systems and Hardware OLPP DPS
诸如NCCL之类的GPU通信库利用流式多处理器(SM)执行数据传输与同步操作,这会拖慢计算与通信的重叠执行,且其性能开销随互连带宽的提升而增大。我们提出“零SM版NCCL”(Zero-SM NCCL),该方案在单个Scale-up域内及跨Scale-up域进行通信时,完全不启动任何SM内核。它通过CUDA流内存操作实现同步,在Scale-up域内借助专用拷贝引擎异步移动数据,并由一个按流排序的CPU代理驱动网络RDMA。该方案支持AllGather、AlltoAll等数据移动类规约操作,同时提供单边通信API——ncclPutSignal与ncclWaitSignal。在GB200 NVL72与B200 NVL8系统上,“零SM版”规约操作的AllGather带宽最高可达传统基于SM内核的NCCL的1.36倍;同时,重叠执行的GEMM运算所受的中位性能下降幅度亦从33%大幅降低至4%,代价是单次操作延迟略有升高。在Megatron-LM训练中,该通信方案使FSDP(全分片数据并行)训练速度最高提升3.5%;而单边put操作则可将流水线并行效率最高提升15%。然而,“零SM通信”并非在所有场景下均具优势。我们通过覆盖多种模型与并行策略的端到端实证研究发现:该方案仅在GPU计算受限型工作负载中的大容量数据传输场景,以及存在较大空泡(bubble)的深层流水线中才真正获益;而在小消息传输或主机(CPU)受限型工作负载中,其性能反而劣于传统方案。我们据此提炼出若干实践指南,用于指导如何在NCCL中启用零SM通信功能,并为将其集成至其他通信库提供设计参考。
GPU communication libraries such as NCCL use Streaming Multiprocessors (SMs) for data movement and synchronization, slowing overlapped computation at a cost that grows with interconnect bandwidth. We present Zero-SM NCCL, which communicates within and across scale-up domains without launching any SM kernel. It synchronizes through CUDA stream memory operations, moves data asynchronously with copy engines within a scale-up domain, and drives network RDMA from a stream-ordered CPU proxy. It supports data-movement collectives such as AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8 systems, Zero-SM collectives reach up to 1.36x higher AllGather bandwidth than NCCL's legacy SM-based kernels and cut the median slowdown of overlapped GEMMs from 33% to 4%, at a higher latency per operation. In Megatron-LM training, they accelerate FSDP by up to 3.5%, and one-sided puts accelerate pipeline parallelism by up to 15%. Zero-SM communication is not a universal win, however. Through an end-to-end study across models and parallelism strategies, we show that it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these findings into guidelines for enabling zero-SM communication in NCCL and building it into other libraries.
许愿