Compass: Large Multilingual Language Model for South-east Asia

大型语言模型在拥有丰富语言资源的语言中（例如英语和中文）表现出了显著的熟练度。然而，当应用于语言资源有限的语言，特别是东南亚语言环境中（例如印度尼西亚语），它们的有效性显著降低。这些语言的语言资源匮乏，导致了训练不足、词汇覆盖范围受限和评估过程具有挑战性的问题。为了应对这些紧迫需求，我们介绍了CompassLLM，这是一个专门针对东南亚语言的大型多语言模型，其主要目的是支持Shopee的开发需求。我们的方法包括几个关键策略。为了逐步提高多语言熟练度，我们实施了一种多阶段预训练策略，结合课程学习，逐渐加强对低资源语言的关注。同时，为了更好地适应低资源人类指令，我们策划和生成了一个高质量的多语言人类指令库，通过受监督的指令微调，最终形成了CompassLLM-SFT模型。最后，为了加强模型与人类偏好行为的一致性，我们采用了直接偏好优化（DPO）原则，获得了CompassLLM-DPO模型。CompassLLM模型的初步评估显示出有希望的结果，我们的模型在各种评估任务中均超过了基准模型，如Vicuna-7b-v1.5、Sealion、Falcon和SeaLLM，经过自动化和人工评估验证，尤其在东南亚语言（如印度尼西亚语）中表现出卓越的性能。

CompassLLM: A Large Multilingual Language Model for Southeast Asian Languages

The paper introduces CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, which addresses the challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes due to the scarcity of linguistic resources for these languages.

The methodology encompasses a multi-stage pre-training strategy integrated with curriculum learning, a repository of high-quality multilingual human instructions, and Direct Preference Optimization (DPO) to reinforce the model's alignment with human preference behaviors. The model surpasses benchmark models across diverse evaluation tasks, with superior performance in Southeast Asian languages, such as Indonesian language.

Some related studies in this field include Vicuna-7b-v1.5, Sealion, Falcon, and SeaLLM.