Barbarians at the Gate: How AI is Upending Systems Research

AI Systems and Hardware CACRR PVDAR
人工智能(AI)正开始通过自动化发现新解决方案,彻底改变我们所熟知的研究方式。对于给定任务,典型的AI驱动方法是:(i)生成一组多样化的解决方案,然后(ii)验证这些方案并选择其中能够解决问题的一个。关键在于,这种方法依赖于一个可靠的验证机制,即能够准确判断某个解决方案是否真正解决了给定问题的评判标准。我们认为,系统研究领域长期以来专注于设计和评估以性能为导向的新算法,因此特别适合采用AI驱动的解决方案探索。原因在于,系统性能问题天然具备可靠的验证手段:解决方案通常可在真实系统或仿真器中实现,而验证过程则简化为在预定义工作负载下运行这些软件组件,并测量其性能表现。我们将这一方法称为“面向系统的AI驱动研究”(AI-Driven Research for Systems, ADRS),它通过不断生成、评估和优化方案来迭代推进研究进程。借助penEvolve——一个现有的开源ADRS实例——我们在多个不同领域开展了案例研究,涵盖多区域云调度中的负载均衡、专家混合模型推理、基于大语言模型的SQL查询以及事务调度等场景。在多个案例中,ADRS发现的算法性能超越了当前最先进的由人工设计的方案(例如,运行时间最高提升5.0倍,或成本降低50%)。我们总结了现有框架下指导算法演进的最佳实践,从提示词设计到评估器构建均有涉及。随后,我们探讨了这一趋势对系统研究社区更广泛的影响:随着AI在算法设计中扮演核心角色,人类研究人员将越来越多地聚焦于问题的定义与战略层面的引导。我们的研究成果既揭示了AI带来的颠覆性潜力,也凸显出在AI时代调整和革新系统研究范式的紧迫性。
Artificial Intelligence (AI) is starting to transform the research process as we know it by automating the discovery of new solutions. Given a task, the typical AI-driven approach is (i) to generate a set of diverse solutions, and then (ii) to verify these solutions and select one that solves the problem. Crucially, this approach assumes the existence of a reliable verifier, i.e., one that can accurately determine whether a solution solves the given problem. We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. We term this approach as AI-Driven Research for Systems (ADRS), which iteratively generates, evaluates, and refines solutions. Using penEvolve, an existing open-source ADRS instance, we present case studies across diverse domains, including load balancing for multi-region cloud scheduling, Mixture-of-Experts inference, LLM-based SQL queries, and transaction scheduling. In multiple instances, ADRS discovers algorithms that outperform state-of-the-art human designs (e.g., achieving up to 5.0x runtime improvements or 50% cost reductions). We distill best practices for guiding algorithm evolution, from prompt design to evaluator construction, for existing frameworks. We then discuss the broader implications for the systems community: as AI assumes a central role in algorithm design, we argue that human researchers will increasingly focus on problem formulation and strategic guidance. Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI.
许愿