Do Methods Support the Claims? Intra-Paper Verification for Peer Review

• URL: http://arxiv.org/abs/2607.26066v1
• Date: Sun, 21 Jun 2026 18:52:53 GMT
• Title: Do Methods Support the Claims? Intra-Paper Verification for Peer Review
• Abstract summary: Existing automated novelty assessment approaches typically compare a paper's claimed contributions against prior literature.<n>Human reviewers, however, frequently challenge novelty claims not because similar ideas already exist, but because the methodological evidence presented in the paper does not adequately support them.<n>We introduce intra-paper claim verification, a framework that evaluates whether novelty claims articulated in a paper are substantiated by the methods used to realize them.
• Score: 0.015552638545879348
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing volume of scientific submissions has motivated interest in using large language models (LLMs) to assist peer review. Existing automated novelty assessment approaches typically compare a paper's claimed contributions against prior literature, implicitly assuming that these contributions are accurately realized in the work itself. Human reviewers, however, frequently challenge novelty claims not because similar ideas already exist, but because the methodological evidence presented in the paper does not adequately support them. This internal mismatch between claimed contributions and methodological realization is rarely examined by current LLM-based review systems. To address this gap, we introduce intra-paper claim verification, a framework that evaluates whether novelty claims articulated in a paper are substantiated by the methods used to realize them. The framework employs an LLM to extract novelty claims from the introduction, retrieve claim-relevant methodological evidence, and assess whether the methods substantiate the stated contributions. Assessment is guided by reviewer-inspired evaluation criteria derived inductively from human peer reviews collected from 182 ICLR 2025 papers. These criteria capture recurring reviewer concerns related to novelty, methodology, clarity, and other issues and are used to generate structured reviewer-style assessments of claim substantiation. We evaluate the framework by comparing LLM-generated review comments against human reviewer concerns on a balanced subset of accepted and rejected papers. Human evaluation demonstrates significant alignment between framework-generated assessments and human reviewer concerns, particularly for novelty-related issues. BERTScore further distinguishes corresponding human-LLM review pairs from mismatched controls, indicating that the framework captures concerns consistent with human reviewer observations.

Related papers

• ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation [54.82704239643649]
  We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans.<n>Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback.<n>We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.
  arXiv  Detail & Related papers  (2026-09-08T17:30:16Z)
• From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent [52.42081442204093]
  We argue that a key limitation is the lack of flexibility to proactively investigate suspicious parts of a paper.<n>We propose ProReviewer, a scientific peer review agent that proactively reviews a paper guided by a maintained, structured review log.<n> Experiments show that ProReviewer with an 8B backbone, trained by supervised fine-tuning and optimized by reinforcement learning, achieves the highest average score across five quality dimensions.
  arXiv  Detail & Related papers  (2026-06-11T13:38:23Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment [17.02516373665209]
  We introduce NovBench, the first large-scale benchmark designed to evaluate large language models' capability to generate novelty evaluations.<n> NovBench comprises 1,684 paper-review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations.<n>We propose a four-dimensional evaluation framework to assess the quality of LLM-generated novelty evaluations.
  arXiv  Detail & Related papers  (2026-04-13T14:35:17Z)
• WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews [19.27332072329702]
  The scientific peer-review process is facing a shortage of human resources due to the rapid growth in the number of submitted papers.<n>We propose a new evaluation metric for scientific review comments that assesses the logical inference between claims and evidence.
  arXiv  Detail & Related papers  (2026-01-24T08:54:25Z)
• When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review [34.067892820832405]
  This paper presents a systematic evaluation of large language models (LLMs) as academic reviewers.<n>Using a curated dataset of 1,441 papers from ICLR 2023 and NeurIPS 2022, we evaluate GPT-5-mini against human reviewers across ratings, strengths, and weaknesses.<n>Our findings show that LLMs consistently inflate ratings for weaker papers while aligning more closely with human judgments on stronger contributions.
  arXiv  Detail & Related papers  (2025-09-12T00:57:50Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Dispute Resolution in Peer Review with Abstract Argumentation and OWL DL [0.12289361708127876]
  This research addresses challenges by applying formal methods from argumentation theory to support transparent and unbiased dispute resolution in peer review.<n>We conceptualize scientific peer review as a single mixed argumentative dispute between manuscript authors and reviewers and formalize it using abstract argumentation frameworks.<n>We validate our approach by annotating a corpus of scientific peer reviews with abstract argumentation frameworks and applying a proof of concept to resolve the disputes.
  arXiv  Detail & Related papers  (2025-07-18T12:21:25Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.