AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation

最近大型语言模型的进展已经改变了机器学习/人工智能的发展，需要重新评估用于检索增强生成系统的AutoML原则。为了解决RAG中的超参数优化和在线适应的挑战，我们提出了AutoRAG-HP框架，将超参数调整制定为在线多臂赌博机（MAB）问题，并引入了一种新的两级分层MAB（Hier-MAB）方法，以便高效地探索大型搜索空间。我们在ALCE-ASQA和自然问题数据集上进行了大量实验，调整了一些超参数，例如前k个检索文档、提示压缩比和嵌入方法。我们的评估结果表明，联合优化这三个超参数的MAB基于在线学习方法，可以在具有突出梯度的搜索空间的情况下，仅使用Grid Search方法所需的大约20%的LLM API调用，就可以实现Recall@5约为0.8。此外，所提出的Hier-MAB方法在更具挑战性的优化场景中优于其他基线。代码将在https://aka.ms/autorag上提供。

AutoRAG-HP framework aims to address the challenges of hyper-parameter optimization and online adaptation in Retrieval-Augmented Generation (RAG) systems.

The framework formulates hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces.

Extensive experiments on tuning hyper-parameters using ALCE-ASQA and Natural Questions datasets demonstrate that MAB-based online learning methods can achieve Recall@5 ~0.8 with only ~20% of the LLM API calls required by the Grid Search approach. The proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.

Recent related works in this field include 'Hyperparameter optimization in deep learning: A systematic review' and 'AutoML: A Survey of the State-of-the-Art'.