PyBench: Evaluating LLM Agent on various real-world coding tasks

Yaolun Zhang ,

Yinxu Pan ,

Yudong Wang ,

Jie Cai ,

Zhi Zheng ,

Guoyang Zeng ,

Zhiyuan Liu

LLM Latent Reasoning CoT Agent Agent Eval Benchmarks

2024年07月23日

LLM Agent配备了一个代码解释器，能够自动解决现实世界中的编码任务，如数据分析和图像编辑。然而，现有的基准主要集中在简单的任务上，例如完成几行代码，或者在存储库级别上进行极其复杂和具体的任务，这两者都不能代表各种日常编码任务。为了填补这一空白，我们介绍了PyBench，这是一个涵盖五个主要类别的现实世界任务的基准，涵盖了10多种文件类型。给定高级用户查询和相关文件，LLM Agent需要通过代码解释器进行一些转换来推理和执行Python代码，然后再做出正式响应以满足用户的需求。成功解决PyBench中的任务需要对各种Python包的全面理解，优秀的推理能力以及将执行的代码的反馈纳入其中的能力。我们的评估表明，目前的开源LLM在这些任务上面临困难。因此，我们对四种数据集进行了分析和实验，证明了PyBench需要综合能力。我们的Fine-tuned 8B大小的模型：PyLlama3在PyBench上取得了令人兴奋的表现，超过了许多33B和70B大小的模型。我们的基准测试、训练数据集和模型都可以在以下网址找到：\href{https://github.com/Mercury7353/PyBench}{https://github.com/Mercury7353/PyBench}。

The LLM Agent, equipped with a code interpreter, is capable of automatically solving real-world coding tasks, such as data analysis and image editing. However, existing benchmarks primarily focus on either simplistic tasks, such as completing a few lines of code, or on extremely complex and specific tasks at the repository level, neither of which are representative of various daily coding tasks. To address this gap, we introduce \textbf{PyBench}, a benchmark encompassing five main categories of real-world tasks, covering more than 10 types of files. Given a high-level user query and related files, the LLM Agent needs to reason and execute Python code via a code interpreter for a few turns before making a formal response to fulfill the user's requirements. Successfully addressing tasks in PyBench demands a robust understanding of various Python packages, superior reasoning capabilities, and the ability to incorporate feedback from executed code. Our evaluations indicate that current open-source LLMs are struggling with these tasks. Hence, we conduct analysis and experiments on four kinds of datasets proving that comprehensive abilities are needed for PyBench. Our fine-tuned 8B size model: \textbf{PyLlama3} achieves an exciting performance on PyBench which surpasses many 33B and 70B size models. Our Benchmark, Training Dataset, and Model are available at: \href{https://github.com/Mercury7353/PyBench}{https://github.com/Mercury7353/PyBench}

许愿