MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models

Dingyao Yu, Kaitao Song, Peiling Lu, Tianyu He, Xu Tan, Wei Ye, Shikun Zhang, Jiang Bian
[Microsoft Research Asia & Peking University]

MusicAgent: 基于大型语言模型的音乐理解和生成的AI智能体

  • MusicAgent是一个利用大语言模型(LLM)如ChatGPT作为控制器的系统，可以自动分析用户请求并调用适当的工具来完成音乐任务。
  • 提供了一个工具包，从Hugging Face、GitHub、Web API等来源收集各种音乐处理工具。
  • 它在任务之间执行标准化的输入输出格式，以实现不同工具之间的无缝协作。
  • 其目标是让用户免于复杂的AI音乐工具，让用户可以专注于创作方面。
  • 关键组件包括任务规划器、工具选择器、任务执行器和响应生成器，所有这些都由LLM提供支持。
  • 它涵盖了多种生成、理解和辅助音乐任务。
  • 代码是模块化和可扩展的，可以轻松集成新工具和自定义。

https://arxiv.org/abs/2310.11954 

动机：构建一个系统来整合和组织音乐相关任务和工具，帮助用户自动分析需求并调用合适的工具来满足需求。

方法：使用大型语言模型(LLM)作为控制器，集成多种音乐相关工具，统一数据格式，实现各种音乐任务的自动化执行。

优势：使用户摆脱复杂的AI音乐工具的细节，通过自动选择最适合的方法，使音乐处理对更广泛的用户群体可用；统一数据格式和协作方式，实现不同平台上工具的无缝协作；高度可扩展，用户可以轻松扩展系统的功能。

通过集成大型语言模型和多种音乐工具，MusicAgent系统实现了音乐相关任务的自动化执行，让用户能更轻松地处理音乐，实现无缝协作和高度可扩展。

AI-empowered music processing is a diverse field that encompasses dozens of tasks, ranging from generation tasks (e.g., timbre synthesis) to comprehension tasks (e.g., music classification). For developers and amateurs, it is very difficult to grasp all of these task to satisfy their requirements in music processing, especially considering the huge differences in the representations of music data and the model applicability across platforms among various tasks. Consequently, it is necessary to build a system to organize and integrate these tasks, and thus help practitioners to automatically analyze their demand and call suitable tools as solutions to fulfill their requirements. Inspired by the recent success of large language models (LLMs) in task automation, we develop a system, named MusicAgent, which integrates numerous musicrelated tools and an autonomous workflow to address user requirements. More specifically, we build 1) toolset that collects tools from diverse sources, including Hugging Face, GitHub, and Web API, etc. 2) an autonomous workflow empowered by LLMs (e.g., ChatGPT) to organize these tools and automatically decompose user requests into multiple sub-tasks and invoke corresponding music tools. The primary goal of this system is to free users from the intricacies of AI-music tools, enabling them to concentrate on the creative aspect. By granting users the freedom to effortlessly combine tools, the system offers a seamless and enriching music experience. The code is available on GitHub1 along with a brief instructional video.


图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除