AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation

• URL: http://arxiv.org/abs/2607.16010v1
• Date: Fri, 17 Jul 2026 14:45:53 GMT
• Title: AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation
• Abstract summary: We evaluate three representative watermarking methods against the Daubert admissibility criteria and the NIST SP 800-86 digital forensic process.<n>We focus on meaning-preserving paraphrase as the attack vector, since it is both legally realistic and difficult to dismiss as evidence tampering.<n>We find that the point-based scoring system, despite working as designed, cannot fully capture forensic uselessness.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Governments are increasingly mandating that LLM-generated content carry watermarks. The EU AI Act calls for markings that are "sufficiently reliable and robust." California's SB 942 requires disclosure that is "permanent or extraordinarily difficult to remove." Both mandates rest on an untested assumption: that watermark detection yields evidence reliable enough for courts. This paper tests that assumption directly. We evaluate three representative LLM watermarking methods -- KGW, Unigram, and the MarkLLM implementation of SynthID-Text -- against the Daubert admissibility criteria and the NIST SP 800-86 digital forensic process. To structure this evaluation, we propose a Forensic Readiness Score (FRS) framework with 12 criteria, three mandatory gates, and a 60-point scoring system. We focus on meaning-preserving paraphrase as the attack vector, since it is both legally realistic and difficult to dismiss as evidence tampering. The results raise serious evidentiary concerns. Out of 846 valid paraphrase runs across 15 diverse prompts per method, every single initially-detected KGW and Unigram text lost its watermark after paraphrasing -- 100% conditional removal. SynthID fared only slightly better at 98.3%. Even before any attack, false-negative rates were already high: 70% for KGW, 83% for Unigram, 80% for SynthID. The SynthID configuration also flagged 5.4% of paraphrased human-written controls as AI-generated and showed an 18.6% paradox rate, with 80% of its own pristine watermarked output landing in the uncertainty deadband. None of the three methods satisfy more than two of five Daubert factors. We also find that the FRS point-based scoring system, despite working as designed, cannot fully capture forensic uselessness -- a limitation worth noting for future framework design. These configurations, as tested, do not meet the evidentiary bar that courts require.

Related papers

• Watermarks Without Verification: AI Text Watermarking After the EU AI Act [49.51124343181601]
  Article 50 of the EU AI Act requires generative AI providers to mark the content their systems produce and ensure it can be detected as AI-generated.<n>Anthropic disclosed that every Claude model released after that date embeds a watermark based on SynthID-Text in all generated text, enabled by default with no user opt-out.<n>We argue that neither the objections nor the assurances can currently be verified and that this unverifiability, rather than watermarking itself, is the substantive governance failure.
  arXiv  Detail & Related papers  (2026-09-09T01:56:17Z)
• Semantic Watermarking with Order-Robust Detection over Sub-sentence Units [21.50298444968968]
  We attack semantic watermarks with order-robust detection over sub-sentence units.<n>The strongest no-box attack we test is an EDA variant adapted to its design, with a 10.8% attack-success rate.<n>A stronger EDA with access to the provider's detector and secret key reaches a 39.7% attack-success rate, compared with 65.5% on k-SemStamp.
  arXiv  Detail & Related papers  (2026-08-27T19:54:17Z)
• Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks [68.64050157343334]
  Existing watermarks preserve provenance under editing, but this same robustness allows an adversary to alter critical content while retaining attribution.<n>We introduce an innovative watermark that jointly provides provenance and tamper evidence.<n>The signals share the same mechanism but use independent keys and different seeding windows over normalized text, making one resilient to edits and the other sensitive to reader-visible changes.
  arXiv  Detail & Related papers  (2026-08-13T01:55:16Z)
• SAMark: A Self-Anchored Text Watermarking with Paragraph-Level Paraphrase Robustness [71.08096624551246]
  robustness to paragraph-level paraphrasing remains difficult because such attacks globally disrupt watermark signals by changing sentence order.<n>We propose SAMark, a self-anchored watermarking framework that removes the dependency on sentence order by establishing a step-independent green region in semantic space.<n> Experimental results show that SAMark achieves up to 90.2% TP@FP1% under typical paragraph-level paraphrasing attacks.
  arXiv  Detail & Related papers  (2026-05-25T12:46:15Z)
• The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems [0.0]
  We apply hallucination prediction to RAG detection, transforming scores into decision sets with finite-sample coverage guarantees.<n>We analyze this failure through the lens of distributional tails, showing that while NLI models achieve acceptable AUC (0.81), the "hardest" hallucinations are semantically indistinguishable from faithful responses.
  arXiv  Detail & Related papers  (2025-12-17T04:22:28Z)
• Cross-Lingual Summarization as a Black-Box Watermark Removal Attack [0.0]
  Cross-lingual summarization attacks constitute a qualitatively stronger attack vector.<n>We show that CLSA reduces watermark detection accuracy more effectively than monolingual paraphrase.<n>Results highlight a practical, low-cost removal pathway that crosses languages and compresses content without visible artifacts.
  arXiv  Detail & Related papers  (2025-10-27T06:55:52Z)
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique [36.96848724920411]
  We introduce a text-preserving watermarking framework that embeds invisible Unicode characters into documents.<n>We experimentally observe a failure rate of less than 0.1% when detecting a reply after fine-tuning with 50 marked documents.<n>No spurious reply was recovered in over 18,000 challenges, corresponding to a 100%TPR@0% FPR.
  arXiv  Detail & Related papers  (2025-10-07T08:34:08Z)
• An Ensemble Framework for Unbiased Language Model Watermarking [60.99969104552168]
  We propose ENS, a novel ensemble framework that enhances the detectability and robustness of unbiased watermarks.<n>ENS sequentially composes multiple independent watermark instances, each governed by a distinct key, to amplify the watermark signal.<n> Empirical evaluations show that ENS substantially reduces the number of tokens needed for reliable detection and increases resistance to smoothing and paraphrasing attacks.
  arXiv  Detail & Related papers  (2025-09-28T19:37:44Z)
• PromptCOS: Towards System Prompt Copyright Auditing for LLMs via Content-level Output Similarity [61.793486262641345]
  We propose PromptCOS, a method for auditing prompt copyright based on content-level output similarity.<n>It embeds watermarks by optimizing the prompt while simultaneously co-optimizing a special verification query and content-level signal marks.<n>For copyright verification, PromptCOS identifies unauthorized usage by comparing the similarity between the suspicious output and the signal mark.
  arXiv  Detail & Related papers  (2025-09-03T08:19:40Z)
• Robustness Assessment and Enhancement of Text Watermarking for Google's SynthID [12.477209114987376]
  SynGuard is a hybrid framework that embeds watermarks at both lexical and semantic levels.<n>We show that SynGuard improves watermark recovery by an average of 11.1% in F1 score compared to SynthID-Text.
  arXiv  Detail & Related papers  (2025-08-27T19:17:09Z)
• Duwak: Dual Watermarks in Large Language Models [49.00264962860555]
  We propose, Duwak, to enhance the efficiency and quality of watermarking by embedding dual secret patterns in both token probability distribution and sampling schemes. We evaluate Duwak extensively on Llama2, against four state-of-the-art watermarking techniques and combinations of them.
  arXiv  Detail & Related papers  (2024-03-12T16:25:38Z)
• Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models [48.409979469683975]
  We introduce the concept of cross-lingual consistency in text watermarking. Preliminary empirical results reveal that current text watermarking technologies lack consistency when texts are translated into various languages. We propose a Cross-lingual Watermark Removal Attack (CWRA) to bypass watermarking.
  arXiv  Detail & Related papers  (2024-02-21T18:48:38Z)
• Who Wrote this Code? Watermarking for Code Generation [53.24895162874416]
  We propose Selective WatErmarking via Entropy Thresholding (SWEET) to detect machine-generated text. Our experiments show that SWEET significantly improves code quality preservation while outperforming all baselines.
  arXiv  Detail & Related papers  (2023-05-24T11:49:52Z)
• Can AI-Generated Text be Reliably Detected? [50.95804851595018]
  Large Language Models (LLMs) perform impressively well in various applications.<n>The potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use.<n>We stress-test the robustness of these AI text detectors in the presence of an attacker.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.