AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation

Saifur Rahman Tamim ,

Amir Labib Khan

NLP TWCT AI Safety / AI Ethics DWAT RA

2026年07月17日

各国政府正日益强制要求大型语言模型（LLM）生成的内容嵌入数字水印。欧盟《人工智能法案》（AI Act）要求水印标记必须“具备足够高的可靠性与鲁棒性”；而加利福尼亚州参议院第942号法案（SB 942）则规定，相关披露信息须“永久有效或极难移除”。这两项法规均基于一项未经验证的假设：水印检测结果所生成的证据，在司法实践中足以被法庭采信。本文即针对这一核心假设展开直接检验。我们选取三种具有代表性的LLM水印方法——KGW、Unigram，以及SynthID-Text在MarkLLM平台上的实现方案——并依据《道伯特规则》（Daubert admissibility criteria）及美国国家标准与技术研究院（NIST）SP 800-86号文件所确立的数字取证流程，对其展开系统性评估。为结构化地开展此项评估，我们提出了一套“取证就绪度评分”（Forensic Readiness Score, FRS）框架：该框架包含12项具体评估指标、3个强制性门槛（mandatory gates），以及总分60分的量化评分体系。我们特别将“语义保持型改写”（meaning-preserving paraphrase）作为主要攻击手段，因其在法律实践中高度真实可信，且难以被简单归类为人为篡改证据的行为。评估结果引发了严重的证据效力担忧。在每种方法下，针对15个主题各异的提示词（prompt）各执行56次有效改写操作（总计846次），所有最初被成功检测出水印的KGW与Unigram文本，在经改写后均彻底丢失水印——条件性移除率达100%；SynthID表现略优，但也高达98.3%。更值得警惕的是，即便在未遭受任何攻击的情况下，三类方法的漏检率（false-negative rate）已居高不下：KGW为70%，Unigram达83%，SynthID亦达80%。此外，SynthID配置还错误地将5.4%经改写的真人撰写文本判定为AI生成；其“悖论率”（paradox rate）达18.6%，即在其自身输出的原始带水印文本中，有80%落入了无法明确判定的“不确定性死区”（uncertainty deadband）。在全部五项《道伯特规则》要素中，没有任何一种方法满足其中两项以上。我们还发现，尽管FRS点数制评分体系本身按设计正常运行，却仍无法充分反映某些方法在法证实践中的根本性失效——这一局限性，对今后同类评估框架的设计具有重要警示意义。综上所述，本研究中所测试的全部水印配置方案，均未能达到司法实践中对证据所要求的证明标准。

Governments are increasingly mandating that LLM-generated content carry watermarks. The EU AI Act calls for markings that are "sufficiently reliable and robust." California's SB 942 requires disclosure that is "permanent or extraordinarily difficult to remove." Both mandates rest on an untested assumption: that watermark detection yields evidence reliable enough for courts. This paper tests that assumption directly. We evaluate three representative LLM watermarking methods -- KGW, Unigram, and the MarkLLM implementation of SynthID-Text -- against the Daubert admissibility criteria and the NIST SP 800-86 digital forensic process. To structure this evaluation, we propose a Forensic Readiness Score (FRS) framework with 12 criteria, three mandatory gates, and a 60-point scoring system. We focus on meaning-preserving paraphrase as the attack vector, since it is both legally realistic and difficult to dismiss as evidence tampering. The results raise serious evidentiary concerns. Out of 846 valid paraphrase runs across 15 diverse prompts per method, every single initially-detected KGW and Unigram text lost its watermark after paraphrasing -- 100% conditional removal. SynthID fared only slightly better at 98.3%. Even before any attack, false-negative rates were already high: 70% for KGW, 83% for Unigram, 80% for SynthID. The SynthID configuration also flagged 5.4% of paraphrased human-written controls as AI-generated and showed an 18.6% paradox rate, with 80% of its own pristine watermarked output landing in the uncertainty deadband. None of the three methods satisfy more than two of five Daubert factors. We also find that the FRS point-based scoring system, despite working as designed, cannot fully capture forensic uselessness -- a limitation worth noting for future framework design. These configurations, as tested, do not meet the evidentiary bar that courts require.

许愿