COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances

我们提供了公开可用的COPAL-ID，这是一种新颖的印度尼西亚语常识推理数据集。与之前的印尼COPA数据集（XCOPA-ID）不同，COPAL-ID融合了印尼本地和文化细节，因此提供了更自然的印尼文化领域内日常因果推理描绘。COPAL-ID由当地人专业撰写，不像翻译的XCOPA-ID那样不太流畅，也没有尴尬的词语。此外，我们提供了标准印尼语和雅加达印尼语两种版本的COPAL-ID，后者是日常对话中常用的方言。COPAL-ID对于现有的开源和最先进的闭源多语言语言模型来说是一个更大的挑战，但对于人类来说却很容易。我们的研究结果表明，即使是当前最好的开源多语言模型也很难表现良好，在COPAL-ID上只能达到65.47%的准确率，远低于缺乏文化细节的XCOPA-ID（79.40%）。尽管GPT-4的得分令人印象深刻，但与其XCOPA-ID得分相比，它仍然表现不佳，而且仍然无法达到人类的表现水平。这表明这些语言模型在理解印尼的本地文化细节方面仍有很大的差距。

The paper aims to present a novel Indonesian language common sense reasoning dataset that incorporates local and cultural nuances, and evaluate the performance of existing multilingual language models on it.

The key idea of the paper is to create a more natural portrayal of day-to-day causal reasoning within the Indonesian cultural sphere by developing a new dataset, COPAL-ID, that is more fluent and free from awkward phrases compared to the translated XCOPA-ID. The paper also evaluates the performance of existing open-sourced and closed state-of-the-art multilingual language models on COPAL-ID.

The paper presents COPAL-ID, a new Indonesian language common sense reasoning dataset that incorporates local and cultural nuances. It also evaluates the performance of existing open-sourced and closed state-of-the-art multilingual language models on COPAL-ID and shows that they struggle to perform well on this dataset, highlighting the need for models that can comprehend local nuances of Indonesian. The experiments were designed to evaluate the accuracy of the models on COPAL-ID and XCOPA-ID datasets. The paper also presents the dataset in both standard Indonesian and in Jakartan Indonesian dialect. The authors have made the dataset publicly available.

Related work includes previous studies on common sense reasoning datasets in other languages, such as English and Chinese, and their evaluation on existing language models. Some of the related papers are 'COPA: A New Dataset for Choice of Plausible Alternatives' and 'XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning'.