BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

LLM MQIA AI Systems and Hardware EDIO
我们推出了 BaseRT——一款专为 Apple Silicon 平台设计、原生基于 Metal 构建的大语言模型(LLM)推理运行时,创下了该硬件平台迄今最高的推理吞吐量纪录。当前主流的运行时系统(包括 llama.cpp 和基于 MLX 的各类框架)均因采用面向通用计算抽象而未适配 Metal 的执行模型及 Apple Silicon 统一内存架构,从而引入了额外开销。BaseRT 通过直接构建于 Metal 之上,深度融合芯片特性:实现算子内核融合(kernel fusion)、针对统一内存架构进行精细化优化,并采用定制化的调度逻辑,从而充分释放了被现有框架所忽视的性能潜力。BaseRT 全面支持所有 Apple M 系列设备,兼容八种量化格式(从 Q2 到 FP16),覆盖多种主流模型家族。本文在 M3 和 M4 Pro 设备上,对 Qwen3、Llama 3.2 和 Gemma 4 三大模型系列分别在 Q4 和 Q8 两种量化精度下进行了系统性评测。实验结果表明,BaseRT 的解码(decode)吞吐量最高可达 llama.cpp 的 1.56 倍、MLX 的 1.35 倍;而对于混合专家(MoE)类模型,其预填充(prefill)阶段的性能优势更为显著。从参数量低于 10 亿的小型模型到高达 300 亿参数的大型模型,BaseRT 均展现出稳定且业界领先的端到端吞吐性能。这些成果重新定义了 Apple Silicon 作为推理平台的能力上限,其意义远超此前报道:随着隐私保护要求日益严格、端到端延迟约束愈发严苛,以及云端推理成本压力持续攀升,AI 推理正加速向终端设备迁移;在此背景下,经过深度性能调优的本地化推理运行时,已成为支撑这一范式转型不可或缺的关键基础设施层。BaseRT 已开源,代码地址为 https://github.com/basecompute/baseRT
We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal's execution model or Apple Silicon's unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon as a more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at https://github.com/basecompute/baseRT
许愿