Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

• URL: http://arxiv.org/abs/2605.21713v1
• Date: Wed, 20 May 2026 20:19:16 GMT
• Title: Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews
• Abstract summary: We propose Sem-Detect, an authorship detection method for peer reviews.<n>It compares a target review against multiple AI-generated reviews of the same paper.<n>It is able to distinguish fully AI reviews from authentic human-written ones.
• Score: 11.21278193949403
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.

Related papers

• AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality [39.33746637947531]
  We survey reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals.<n>We evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset.
  arXiv  Detail & Related papers  (2026-08-04T12:34:47Z)
• Show, Don't TELL: Explainable AI-Generated Text Detection [0.42970700836450487]
  We develop a novel architecture, TELL, that bakes explainability from the ground-up.<n>We train TELL on a custom SFT dataset of domain-specific authorship annotations.<n>We report a high (mean 72.3%) win-rate on annotation concreteness, falsifiability, coherence, plausibility and grounding.
  arXiv  Detail & Related papers  (2026-05-27T03:47:25Z)
• Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
  This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
  arXiv  Detail & Related papers  (2026-05-04T22:41:04Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• PeerPrism: Peer Evaluation Expertise vs Review-writing AI [12.533035088439975]
  We introduce PeerPrism, a benchmark of 20,690 peer reviews explicitly designed to disentangle idea provenance from text provenance.<n>We benchmark state-of-the-art LLM text detection methods on PeerPrism.<n>Our results show that current detection methods conflate surface realization with intellectual contribution.
  arXiv  Detail & Related papers  (2026-04-16T00:59:55Z)
• Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable [17.525659482226875]
  A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers.<n>We assemble a dataset of peer reviews simulating multiple levels of human-AI collaboration, and evaluate five state-of-the-art detectors.<n>Our analysis shows that all detectors misclassify a non-trivial fraction of LLM-polished reviews as AI-generated, risking false accusations of academic misconduct.
  arXiv  Detail & Related papers  (2026-03-20T19:28:34Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review [6.20631177269082]
  A new risk to the peer review process is that negligent reviewers will rely on large language models (LLMs) to review a paper.<n>We introduce a comprehensive dataset containing a total of 788,984 AI-written peer reviews paired with corresponding human reviews.<n>We use this new resource to evaluate the ability of 18 existing AI text detection algorithms to distinguish between peer reviews fully written by humans and different state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-02-26T23:04:05Z)
• Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing [55.2480439325792]
  This study systematically evaluations twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation dataset.<n>Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models.
  arXiv  Detail & Related papers  (2025-02-21T18:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.