Technical Report on the Checkfor.ai AI-Generated Text Classifier

AI Safety / AI Ethics DWAT AFBD
我们推出了CheckforAI文本分类器,这是一个基于Transformer的神经网络,训练目的是区分由大型语言模型编写的文本和由人类编写的文本。在包括学生写作、创意写作、科学写作、书籍、百科全书、新闻、电子邮件、科学论文、短格式问答等十个文本领域和8个开源和闭源大型语言模型的综合基准测试中,CheckforAI的表现优于零-shot方法(例如DetectGPT)以及领先的商业AI检测工具,误差率低至其九倍以下。我们提出了一种训练算法,即使用合成镜像的硬负样本挖掘,使我们的分类器在评论等高数据领域实现了数量级更低的假阳性率。最后,我们展示了CheckforAI不对非英语母语人士存在偏见,并且在训练期间未曾涉及的领域和模型上具有普适性。
We present the CheckforAI text classifier, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. CheckforAI outperforms zero-shot methods such as DetectGPT as well as leading commercial AI detection tools with over 9 times lower error rates on a comprehensive benchmark comprised of ten text domains (student writing, creative writing, scientific writing, books, encyclopedias, news, email, scientific papers, short-form Q&A) and 8 open- and closed-source large language models. We propose a training algorithm, hard negative mining with synthetic mirrors, that enables our classifier to achieve orders of magnitude lower false positive rates on high-data domains such as reviews. Finally, we show that CheckforAI is not biased against nonnative English speakers and generalizes to domains and models unseen during training.
许愿