AutoTest: Evolutionary Code Solution Selection with Test Cases

随着代码生成技术的发展，从多个候选方案中选择正确的代码解决方案已成为一项关键任务。本研究提出了一种新颖的技术AutoTest，它将自动化测试用例生成与代码解决方案执行相结合，利用进化遗传算法优化选择过程。首先，AutoTest利用大型预训练语言模型（如codegen-16B、code-davinci-002和incoder-6B）提供代码解决方案及其相应的测试用例。然后，通过执行代码解决方案并在测试用例上评估其性能，形成共识集。通过基于进化遗传算法的选择、变异和交叉机制以及alpha和beta参数的调整，实现了细粒度排名。最后，选择最佳代码解决方案。AutoTest在HumanEval基准测试上展现出显著的性能提升。HumanEval数据集包含164个编程问题，AutoTest在pass@1分数方面比基线方法提高了约10%。

AutoTest: A Genetic Algorithm for Optimizing Code Solution Selection Based on Automated Test Case Generation

AutoTest combines automated test case generation with code solution execution to optimize the selection process using an evolutionary genetic algorithm, achieving fine-grained ranking through selection, mutation, and crossover mechanisms.

AutoTest utilizes large pre-trained language models to provide code solutions and their corresponding test cases, and demonstrates significant performance improvements on the HumanEval benchmark test. The experiment is well-designed and the paper provides open-source code and data for reproducibility. Further research could explore the application of AutoTest in other programming languages and domains.

Recent related work includes Neural-Guided Deductive Search for Real-Time Program Synthesis and Evolutionary Synthesis of Deep Neural Networks with Network Morphism Operators.