CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection

• URL: http://arxiv.org/abs/2509.04460v1
• Date: Thu, 28 Aug 2025 06:03:11 GMT
• Title: CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
• Authors: Yihan Chen, Jiawei Chen, Guozhao Mo, Xuanang Chen, Ben He, Xianpei Han, Le Sun,
• Abstract summary: We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
• Score: 60.52240468810558
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The growing integration of large language models (LLMs) into the peer review process presents potential risks to the fairness and reliability of scholarly evaluation. While LLMs offer valuable assistance for reviewers with language refinement, there is growing concern over their use to generate substantive review content. Existing general AI-generated text detectors are vulnerable to paraphrasing attacks and struggle to distinguish between surface language refinement and substantial content generation, suggesting that they primarily rely on stylistic cues. When applied to peer review, this limitation can result in unfairly suspecting reviews with permissible AI-assisted language enhancement, while failing to catch deceptively humanized AI-generated reviews. To address this, we propose a paradigm shift from style-based to content-based detection. Specifically, we introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews, covering six distinct modes of human-AI collaboration. Furthermore, we develop CoCoDet, an AI review detector via a multi-task learning framework, designed to achieve more accurate and robust detection of AI involvement in review content. Our work offers a practical foundation for evaluating the use of LLMs in peer review, and contributes to the development of more precise, equitable, and reliable detection methods for real-world scholarly applications. Our code and data will be publicly available at https://github.com/Y1hanChen/COCONUTS.

Related papers

• DAMASHA: Detecting AI in Mixed Adversarial Texts via Segmentation with Human-interpretable Attribution [20.178134447843092]
  In the age of advanced large language models, the boundaries between human and AI-generated text are becoming increasingly blurred.<n>We introduce a novel framework, called Info-Mask for mixed authorship detection that integrates stylometric cues, perplexity-driven signals, and structured boundary modeling.<n>Our findings highlight both the promise and limitations of adversarially robust, interpretable mixed-authorship detection.
  arXiv  Detail & Related papers  (2025-12-04T14:21:42Z)
• The Impact of LLM-Generated Reviews on Recommender Systems: Textual Shifts, Performance Effects, and Strategic Platform Control [0.4078247440919473]
  The rise of generative AI technologies is reshaping content-based recommender systems (RSes)<n>This study examines how the introduction of AI-generated reviews influences RS performance and business outcomes.
  arXiv  Detail & Related papers  (2025-11-02T09:06:47Z)
• Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection [71.59834293521074]
  We develop a framework to distinguish between human-authored and machine-generated text.<n>Our method achieves 98.3% AUROC and AUPR with only 8.9% FPR95 on DeepFake dataset.<n>Code, pretrained weights, and demo will be released.
  arXiv  Detail & Related papers  (2025-10-07T08:14:45Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review [6.20631177269082]
  A new risk to the peer review process is that negligent reviewers will rely on large language models (LLMs) to review a paper.<n>We introduce a comprehensive dataset containing a total of 788,984 AI-written peer reviews paired with corresponding human reviews.<n>We use this new resource to evaluate the ability of 18 existing AI text detection algorithms to distinguish between peer reviews fully written by humans and different state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-02-26T23:04:05Z)
• Who Writes the Review, Human or AI? [0.36498648388765503]
  This study proposes a methodology to accurately distinguish AI-generated and human-written book reviews. Our approach utilizes transfer learning, enabling the model to identify generated text across different topics. The experimental results demonstrate that it is feasible to detect the original source of text, achieving an accuracy rate of 96.86%.
  arXiv  Detail & Related papers  (2024-05-30T17:38:44Z)
• PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models [72.57329554067195]
  ProxyQA is an innovative framework dedicated to assessing longtext generation. It comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. It assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions.
  arXiv  Detail & Related papers  (2024-01-26T18:12:25Z)
• Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey [97.33926242130732]
  Large Language Models (LLMs) have revolutionized the domain of natural language processing (NLP) with remarkable capabilities of generating human-like text responses. Despite these advancements, several works in the existing literature have raised serious concerns about the potential misuse of LLMs. To address these concerns, a consensus among the research community is to develop algorithmic solutions to detect AI-generated text.
  arXiv  Detail & Related papers  (2023-10-23T18:11:32Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.