Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code

• URL: http://arxiv.org/abs/2502.18851v1
• Date: Wed, 26 Feb 2025 05:46:13 GMT
• Title: Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code
• Authors: Jungin Kim, Shinwoo Park, Yo-Sub Han,
• Abstract summary: Code watermarking identifies AI-generated code by embedding patterns into the code during generation.<n>Existing methods often modify tokens that are critical for program logic, such as keywords in conditional expressions.<n>We present STONE, a method that preserves functional integrity by selectively inserting watermarks only into non-syntax tokens.
• Score: 4.608510640547953
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Code watermarking identifies AI-generated code by embedding patterns into the code during generation. Effective watermarking requires meeting two key conditions: the watermark should be reliably detectable, and the code should retain its original functionality. However, existing methods often modify tokens that are critical for program logic, such as keywords in conditional expressions or operators in arithmetic computations. These modifications can cause syntax errors or functional failures, limiting the practical use of watermarking. We present STONE, a method that preserves functional integrity by selectively inserting watermarks only into non-syntax tokens. By excluding tokens essential for code execution, STONE minimizes the risk of functional degradation. In addition, we introduce CWEM, a comprehensive evaluation metric that evaluates watermarking techniques based on correctness, detectability, and naturalness. While correctness and detectability have been widely used, naturalness remains underexplored despite its importance. Unnatural patterns can reveal the presence of a watermark, making it easier for adversaries to remove. We evaluate STONE using CWEM and compare its performance with the state-of-the-art approach. The results show that STONE achieves an average improvement of 7.69% in CWEM across Python, C++, and Java. Our code is available in https://github.com/inistory/STONE-watermarking/.

Related papers

• Is The Watermarking Of LLM-Generated Code Robust? [5.48277165801539]
  We show that watermarking techniques are significantly more fragile in code-based contexts.<n>Specifically, we show that simple semantic-preserving transformations, such as variable renaming and dead code insertion, can effectively erase watermarks.
  arXiv  Detail & Related papers  (2024-03-24T21:41:29Z)
• Improving the Generation Quality of Watermarked Large Language Models via Word Importance Scoring [81.62249424226084]
  Token-level watermarking inserts watermarks in the generated texts by altering the token probability distributions. This watermarking algorithm alters the logits during generation, which can lead to a downgraded text quality. We propose to improve the quality of texts generated by a watermarked language model by Watermarking with Importance Scoring (WIS)
  arXiv  Detail & Related papers  (2023-11-16T08:36:00Z)
• A Robust Semantics-based Watermark for Large Language Model against Paraphrasing [50.84892876636013]
  Large language models (LLMs) have show great ability in various natural language tasks. There are concerns that LLMs are possible to be used improperly or even illegally. We propose a semantics-based watermark framework SemaMark.
  arXiv  Detail & Related papers  (2023-11-15T06:19:02Z)
• A Semantic Invariant Robust Watermark for Large Language Models [27.522264953691746]
  Prior watermark algorithms face a trade-off between attack robustness and security robustness. This is because the watermark logits for a token are determined by a certain number of preceding tokens. We propose a semantic invariant watermarking method for LLMs that provides both attack robustness and security robustness.
  arXiv  Detail & Related papers  (2023-10-10T06:49:43Z)
• An Unforgeable Publicly Verifiable Watermark for Large Language Models [84.2805275589553]
  Current watermark detection algorithms require the secret key used in the watermark generation process, making them susceptible to security breaches and counterfeiting during public detection. We propose an unforgeable publicly verifiable watermark algorithm named UPV that uses two different neural networks for watermark generation and detection, instead of using the same key at both stages.
  arXiv  Detail & Related papers  (2023-07-30T13:43:27Z)
• Who Wrote this Code? Watermarking for Code Generation [53.24895162874416]
  We propose Selective WatErmarking via Entropy Thresholding (SWEET) to detect machine-generated text. Our experiments show that SWEET significantly improves code quality preservation while outperforming all baselines.
  arXiv  Detail & Related papers  (2023-05-24T11:49:52Z)
• Towards Tracing Code Provenance with Code Watermarking [37.41260851333952]
  We propose CodeMark, a watermarking system that hides bit strings into variables respecting the natural and operational semantics of the code. For naturalness, we introduce a contextual watermarking scheme to generate watermarked variables more coherent in the context atop graph neural networks. We show CodeMark outperforms the SOTA watermarking systems with a better balance of the watermarking requirements.
  arXiv  Detail & Related papers  (2023-05-21T13:53:12Z)
• Watermarking Text Generated by Black-Box Language Models [103.52541557216766]
  A watermark-based method was proposed for white-box LLMs, allowing them to embed watermarks during text generation. A detection algorithm aware of the list can identify the watermarked text. We develop a watermarking framework for black-box language model usage scenarios.
  arXiv  Detail & Related papers  (2023-05-14T07:37:33Z)
• A Watermark for Large Language Models [84.95327142027183]
  We propose a watermarking framework for proprietary language models. The watermark can be embedded with negligible impact on text quality. It can be detected using an efficient open-source algorithm without access to the language model API or parameters.
  arXiv  Detail & Related papers  (2023-01-24T18:52:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.