WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews

• URL: http://arxiv.org/abs/2601.17377v1
• Date: Sat, 24 Jan 2026 08:54:25 GMT
• Title: WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews
• Abstract summary: The scientific peer-review process is facing a shortage of human resources due to the rapid growth in the number of submitted papers.<n>We propose a new evaluation metric for scientific review comments that assesses the logical inference between claims and evidence.
• Score: 19.27332072329702
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The scientific peer-review process is facing a shortage of human resources due to the rapid growth in the number of submitted papers. The use of language models to reduce the human cost of peer review has been actively explored as a potential solution to this challenge. A method has been proposed to evaluate the level of substantiation in scientific reviews in a manner that is interpretable by humans. This method extracts the core components of an argument, claims and evidence, and assesses the level of substantiation based on the proportion of claims supported by evidence. The level of substantiation refers to the extent to which claims are based on objective facts. However, when assessing the level of substantiation, simply detecting the presence or absence of supporting evidence for a claim is insufficient; it is also necessary to accurately assess the logical inference between a claim and its evidence. We propose a new evaluation metric for scientific review comments that assesses the logical inference between claims and evidence. Experimental results show that the proposed method achieves a higher correlation with human scores than conventional methods, indicating its potential to better support the efficiency of the peer-review process.

Related papers

• Do Methods Support the Claims? Intra-Paper Verification for Peer Review [0.015552638545879348]
  Existing automated novelty assessment approaches typically compare a paper's claimed contributions against prior literature.<n>Human reviewers, however, frequently challenge novelty claims not because similar ideas already exist, but because the methodological evidence presented in the paper does not adequately support them.<n>We introduce intra-paper claim verification, a framework that evaluates whether novelty claims articulated in a paper are substantiated by the methods used to realize them.
  arXiv  Detail & Related papers  (2026-06-21T18:52:53Z)
• From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent [52.42081442204093]
  We argue that a key limitation is the lack of flexibility to proactively investigate suspicious parts of a paper.<n>We propose ProReviewer, a scientific peer review agent that proactively reviews a paper guided by a maintained, structured review log.<n> Experiments show that ProReviewer with an 8B backbone, trained by supervised fine-tuning and optimized by reinforcement learning, achieves the highest average score across five quality dimensions.
  arXiv  Detail & Related papers  (2026-06-11T13:38:23Z)
• Measuring AI Reasoning: A Guide for Researchers [19.301617369098135]
  We argue that reasoning should be assessed through evidence of adaptive, multi-step search rather than final-answer accuracy alone.<n>We show that single forward passes in scalable architectures are structurally limited in their ability to realize such variable-depth computation.<n>We argue for a shift toward process-based evaluation, in which reasoning is assessed through the faithfulness and validity of intermediate reasoning traces as first-class evaluation targets.
  arXiv  Detail & Related papers  (2026-05-04T10:42:26Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem [87.30601926271864]
  InnoEval is a deep innovation evaluation framework designed to emulate human-level idea assessment.<n>We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources.<n>We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval.
  arXiv  Detail & Related papers  (2026-02-16T00:40:31Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Dispute Resolution in Peer Review with Abstract Argumentation and OWL DL [0.12289361708127876]
  This research addresses challenges by applying formal methods from argumentation theory to support transparent and unbiased dispute resolution in peer review.<n>We conceptualize scientific peer review as a single mixed argumentative dispute between manuscript authors and reviewers and formalize it using abstract argumentation frameworks.<n>We validate our approach by annotating a corpus of scientific peer reviews with abstract argumentation frameworks and applying a proof of concept to resolve the disputes.
  arXiv  Detail & Related papers  (2025-07-18T12:21:25Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking [11.300523252168327]
  Evtextsuperscript2R combines the strengths of reference-based evaluation and verdict-level proxy scoring.<n>Evtextsuperscript2R consistently outperforms existing scoring approaches in accuracy and robustness.
  arXiv  Detail & Related papers  (2024-11-08T07:05:06Z)
• Demonstrative Evidence and the Use of Algorithms in Jury Trials [0.0]
  We investigate how the use of bullet comparison algorithms and demonstrative evidence may affect juror perceptions of reliability, credibility, and understanding of expert witnesses and presented evidence. We find that individuals overwhelmingly provided high Likert scale ratings in reliability, credibility, and scientificity regardless of experimental condition.
  arXiv  Detail & Related papers  (2023-11-18T04:21:01Z)
• From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification [118.03466985807331]
  We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence.<n>We introduce the feedback-based evidence retriever(FER) that optimize the evidence retrieval process by incorporating feedback from the claim verifier.
  arXiv  Detail & Related papers  (2023-10-18T02:59:38Z)
• Generating Scientific Claims for Zero-Shot Scientific Fact Checking [54.62086027306609]
  Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data. We propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences. We also demonstrate its usefulness in zero-shot fact checking for biomedical claims.
  arXiv  Detail & Related papers  (2022-03-24T11:29:20Z)
• Assessing the Sufficiency of Arguments through Conclusion Generation [15.69811781628757]
  An argument whose premises make its conclusion rationally worthy is called sufficient in argument quality research. In this paper, we hypothesize that the conclusion of a sufficient argument can be generated from its premises. Our best model achieves variant an F1-score of.885, outperforming the previous state-of-the-art and being on par with human experts.
  arXiv  Detail & Related papers  (2021-10-26T08:47:53Z)
• On the Interaction of Belief Bias and Explanations [4.211128681972148]
  We provide an overview of belief bias, its role in human evaluation, and ideas for NLP practitioners on how to account for it. We show that conclusions about the highest performing methods change when introducing such controls, pointing to the importance of accounting for belief bias in evaluation.
  arXiv  Detail & Related papers  (2021-06-29T12:49:42Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.