Pretraining Data Can Be Poisoned through Computational Propaganda

Victoria Graf ,

Hannaneh Hajishirzi ,

Noah A. Smith ,

David Kohlbrenner ,

Kyle Lo

LLM Other LLM AI Safety / AI Ethics MBDSF RA

对预训练数据实施投毒攻击，可能向语言模型引入难以检测与缓解的有害行为。以往关于预训练数据投毒的研究，大多依赖维基百科等已被广泛采用的数据源；而这类数据源既无法体现真实预训练语料所具有的海量规模与高度异质性，也忽略了被投毒数据与数据筛选流程之间的交互影响。我们证明：除上述受限场景外，利用现有、面向全网规模的内容注入机制——即公开讨论界面（public discussion interfaces），同样可成功实施针对预训练数据的投毒攻击。此外，为评估网络爬取及后续数据筛选流程完成后，恶意内容是否仍保留在训练数据中，我们提出一种新型分析方法“半衰期”（HalfLife），用于估算基于网络爬虫构建的语言模型训练数据中对抗性内容的实际留存比例。我们运用“半衰期”方法，系统探究了借助开放型讨论界面在全网尺度上对预训练语料库实施投毒攻击的可行性。我们的分析表明：准确估计投毒内容是否最终进入预训练数据，具有至关重要的意义；同时，本研究首次确立第三方网页内容可作为一种切实可行的攻击路径，用以危害语言模型的预训练过程。

Poisoning pretraining data can introduce harmful behaviors to LMs that are difficult to detect and mitigate. Prior work on poisoning pretraining data has largely exploited established data sources such as Wikipedia, which do not represent the large scale and heterogeneity typical of pretraining corpora, and has ignored the interaction between poisoned data and data curation pipelines. We demonstrate that poisoning attacks on pretraining data are feasible beyond this limited setting through an existing web-scale content injection mechanism: public discussion interfaces. Additionally, to measure whether malicious content is included after web crawling and data curation, we introduce HalfLife, a novel analysis for estimating adversarial content inclusion in web-crawl based LM training data. We use HalfLife to explore the feasibility of poisoning pretraining corpora at web scale through open discussion interfaces. Our analysis demonstrates the importance of estimating whether poison injections are included in pretraining data, and establishes third-party webpage content as a possible vector for attacking language model pretraining.

许愿