The Feasibility of Topic-Based Watermarking on Academic Peer Reviews

• URL: http://arxiv.org/abs/2505.21636v1
• Date: Tue, 27 May 2025 18:09:27 GMT
• Title: The Feasibility of Topic-Based Watermarking on Academic Peer Reviews
• Authors: Alexander Nemecek, Yuzhou Jiang, Erman Ayday,
• Abstract summary: We evaluate topic-based watermarking (TBW) for large language models (LLMs)<n>Our results show that TBW maintains review quality relative to non-watermarked outputs, while demonstrating strong to paraphrasing-based evasion.
• Score: 46.71493672772134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly integrated into academic workflows, with many conferences and journals permitting their use for tasks such as language refinement and literature summarization. However, their use in peer review remains prohibited due to concerns around confidentiality breaches, hallucinated content, and inconsistent evaluations. As LLM-generated text becomes more indistinguishable from human writing, there is a growing need for reliable attribution mechanisms to preserve the integrity of the review process. In this work, we evaluate topic-based watermarking (TBW), a lightweight, semantic-aware technique designed to embed detectable signals into LLM-generated text. We conduct a comprehensive assessment across multiple LLM configurations, including base, few-shot, and fine-tuned variants, using authentic peer review data from academic conferences. Our results show that TBW maintains review quality relative to non-watermarked outputs, while demonstrating strong robustness to paraphrasing-based evasion. These findings highlight the viability of TBW as a minimally intrusive and practical solution for enforcing LLM usage in peer review.

Related papers

• In-Context Watermarks for Large Language Models [71.29952527565749]
  In-Context Watermarking (ICW) embeds watermarks into generated text solely through prompt engineering.<n>We investigate four ICW strategies at different levels of granularity, each paired with a tailored detection method.<n>Our experiments validate the feasibility of ICW as a model-agnostic, practical watermarking approach.
  arXiv  Detail & Related papers  (2025-05-22T17:24:51Z)
• Detecting LLM-Generated Peer Reviews [37.51215252353345]
  The rise of large language models (LLMs) has introduced concerns that some reviewers may rely on these tools to generate reviews rather than writing them independently.<n>We consider the approach of performing indirect prompt injection via the paper's PDF, prompting the LLM to embed a covert watermark in the generated review.<n>We introduce watermarking schemes and hypothesis tests that control the family-wise error rate across multiple reviews, achieving higher statistical power than standard corrections.
  arXiv  Detail & Related papers  (2025-03-20T01:11:35Z)
• Large Language Models Penetration in Scholarly Writing and Peer Review [43.600778691549706]
  We evaluate the penetration of Large Language Models across academic perspectives and dimensions.<n>Our experiments demonstrate the effectiveness of textttLLMetrica, revealing the increasing role of LLMs in scholarly processes.<n>These findings emphasize the need for transparency, accountability, and ethical practices in LLM usage to maintain academic credibility.
  arXiv  Detail & Related papers  (2025-02-16T16:37:34Z)
• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review [66.73247554182376]
  Large language models (LLMs) have led to their integration into peer review.<n>The unchecked adoption of LLMs poses significant risks to the integrity of the peer review system.<n>We show that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings.
  arXiv  Detail & Related papers  (2024-12-02T16:55:03Z)
• Streamlining the review process: AI-generated annotations in research manuscripts [0.5735035463793009]
  This study explores the potential of integrating Large Language Models (LLMs) into the peer-review process to enhance efficiency without compromising effectiveness.<n>We focus on manuscript annotations, particularly excerpt highlights, as a potential area for AI-human collaboration.<n>This paper introduces AnnotateGPT, a platform that utilizes GPT-4 for manuscript review, aiming to improve reviewers' comprehension and focus.
  arXiv  Detail & Related papers  (2024-11-29T23:26:34Z)
• Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? [62.72729485995075]
  We investigate the effectiveness of watermarking as a deterrent against the generation of copyrighted texts.<n>We find that watermarking adversely affects the success rate of Membership Inference Attacks (MIAs)<n>We propose an adaptive technique to improve the success rate of a recent MIA under watermarking.
  arXiv  Detail & Related papers  (2024-07-24T16:53:09Z)
• Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions [62.0123588983514]
  Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields. We reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources.
  arXiv  Detail & Related papers  (2024-06-09T08:24:17Z)
• FABLES: Evaluating faithfulness and content selection in book-length summarization [55.50680057160788]
  In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on book-length documents. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
  arXiv  Detail & Related papers  (2024-04-01T17:33:38Z)
• SIFiD: Reassess Summary Factual Inconsistency Detection with LLM [27.392514180175283]
  This study reassesses summary inconsistency detection with Large Language Models (LLMs) We propose SIFiD (Summary Inconsistency Detection with Filtered Document) that identify key sentences within documents by either employing natural language inference or measuring semantic similarity between summaries and documents.
  arXiv  Detail & Related papers  (2024-03-12T11:41:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.