作者：F Gilardi, M Alizadeh, M Kubli
[University of Zurich]

总结:
ChatGPT在几个标注任务中的性能超过人工众包工人，且成本比MTurk便宜20倍。

要点:

  1. ChatGPT在几个标注任务中的性能超过人工众包工人，包括相关性、立场、主题和框架检测
  2. ChatGPT的译码器一致性超过人工众包工人和训练有素的标注者
  3. ChatGPT的标注成本约为MTurk的20分之一
  4. 未来可以对ChatGPT进行多语言、多文本类型、少样本学习和半自动化数据标注系统等方面的探究。

https://arxiv.org/abs/2303.15056

Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT’s intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003—about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.

图片

 

 

内容中包含的图片若涉及版权问题，请及时与我们联系删除