A Reinforcement Learning Framework for Robust and Secure LLM Watermarking

• URL: http://arxiv.org/abs/2510.21053v1
• Date: Thu, 23 Oct 2025 23:53:03 GMT
• Title: A Reinforcement Learning Framework for Robust and Secure LLM Watermarking
• Abstract summary: We propose an end-to-end RL framework for robust and secure watermarking.<n>Our approach adopts an anchoring mechanism for reward terms to ensure stable training and introduces additional regularization terms to prevent reward hacking.<n>Our method achieves a state-of-the-art trade-off across all criteria, with notable improvements in resistance to spoofing attacks without degrading other criteria.
• Score: 40.90513051455007
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Watermarking has emerged as a promising solution for tracing and authenticating text generated by large language models (LLMs). A common approach to LLM watermarking is to construct a green/red token list and assign higher or lower generation probabilities to the corresponding tokens, respectively. However, most existing watermarking algorithms rely on heuristic green/red token list designs, as directly optimizing the list design with techniques such as reinforcement learning (RL) comes with several challenges. First, desirable watermarking involves multiple criteria, i.e., detectability, text quality, robustness against removal attacks, and security against spoofing attacks. Directly optimizing for these criteria introduces many partially conflicting reward terms, leading to an unstable convergence process. Second, the vast action space of green/red token list choices is susceptible to reward hacking. In this paper, we propose an end-to-end RL framework for robust and secure LLM watermarking. Our approach adopts an anchoring mechanism for reward terms to ensure stable training and introduces additional regularization terms to prevent reward hacking. Experiments on standard benchmarks with two backbone LLMs show that our method achieves a state-of-the-art trade-off across all criteria, with notable improvements in resistance to spoofing attacks without degrading other criteria. Our code is available at https://github.com/UCSB-NLP-Chang/RL-watermark.

Related papers

• Character-Level Perturbations Disrupt LLM Watermarks [64.60090923837701]
  We formalize the system model for Large Language Model (LLM) watermarking.<n>We characterize two realistic threat models constrained on limited access to the watermark detector.<n>We demonstrate character-level perturbations are significantly more effective for watermark removal under the most restrictive threat model.<n> Experiments confirm the superiority of character-level perturbations and the effectiveness of the Genetic Algorithm (GA) in removing watermarks under realistic constraints.
  arXiv  Detail & Related papers  (2025-09-11T02:50:07Z)
• Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking [48.26359966929394]
  Invisible Entropy (IE) is a watermarking paradigm designed to enhance both safety and efficiency.<n>IE reduces parameter size by 99% while achieving performance on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-20T09:19:06Z)
• Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code [8.371877324812505]
  We present STONE, a syntax-aware watermarking method that embeds watermarks only in non-syntactic tokens.<n>Across Python, C++, and Java, STONE preserves correctness, sustains strong detectability, and achieves balanced performance with minimal overhead.
  arXiv  Detail & Related papers  (2025-02-26T05:46:13Z)
• Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning [58.57194301645823]
  Large language models (LLMs) are increasingly integrated into real-world personalized applications.<n>The valuable and often proprietary nature of the knowledge bases used in RAG introduces the risk of unauthorized usage by adversaries.<n>Existing methods that can be generalized as watermarking techniques to protect these knowledge bases typically involve poisoning or backdoor attacks.<n>We propose name for harmless' copyright protection of knowledge bases.
  arXiv  Detail & Related papers  (2025-02-10T09:15:56Z)
• Large Language Model Watermark Stealing With Mixed Integer Programming [51.336009662771396]
  Large Language Model (LLM) watermark shows promise in addressing copyright, monitoring AI-generated text, and preventing its misuse. Recent research indicates that watermarking methods using numerous keys are susceptible to removal attacks. We propose a novel green list stealing attack against the state-of-the-art LLM watermark scheme.
  arXiv  Detail & Related papers  (2024-05-30T04:11:17Z)
• Watermark Stealing in Large Language Models [2.1165011830664673]
  We show that querying the API of the watermarked LLM to approximately reverse-engineer a watermark enables practical spoofing attacks. We are the first to propose an automated WS algorithm and use it in the first comprehensive study of spoofing and scrubbing in realistic settings.
  arXiv  Detail & Related papers  (2024-02-29T17:12:39Z)
• A Semantic Invariant Robust Watermark for Large Language Models [27.522264953691746]
  Prior watermark algorithms face a trade-off between attack robustness and security robustness. This is because the watermark logits for a token are determined by a certain number of preceding tokens. We propose a semantic invariant watermarking method for LLMs that provides both attack robustness and security robustness.
  arXiv  Detail & Related papers  (2023-10-10T06:49:43Z)
• SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation [72.10931780019297]
  Existing watermarking algorithms are vulnerable to paraphrase attacks because of their token-level design. We propose SemStamp, a robust sentence-level semantic watermarking algorithm based on locality-sensitive hashing (LSH) Experimental results show that our novel semantic watermark algorithm is not only more robust than the previous state-of-the-art method on both common and bigram paraphrase attacks, but also is better at preserving the quality of generation.
  arXiv  Detail & Related papers  (2023-10-06T03:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.