When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews

• URL: http://arxiv.org/abs/2605.10171v1
• Date: Mon, 11 May 2026 08:20:58 GMT
• Title: When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
• Abstract summary: RevCI is an expert-annotated benchmark of peer-review pairs with evidence-level contradiction annotations with graded intensity labels.<n> IMPACT is a structured multi-agent framework that integrates aspect-conditioned evidence extraction, deliberative reasoning, and adjudication.<n>TIDE is a small language model that predicts contradiction evidence and intensity in a single forward pass.
• Score: 25.43097618275061
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific peer reviews frequently contain conflicting expert judgments, and the increasing scale of conference submissions makes it challenging for Area Chairs and editors to reliably identify and interpret such disagreements. Existing approaches typically frame reviewer disagreement as binary contradiction detection over isolated sentence pairs, abstracting away the review-level context and obscuring differences in the severity of evaluative conflict. In this work, we introduce a fine-grained formulation of reviewer contradiction analysis that operates over full peer reviews by explicitly identifying contradiction evidence spans and assigning graded disagreement intensity scores. To support this task, we present RevCI, an expert-annotated benchmark of peer-review pairs with evidence-level contradiction annotations with graded intensity labels. We further propose IMPACT, a structured multi-agent framework that integrates aspect-conditioned evidence extraction, deliberative reasoning, and adjudication to model reviewer contradictions and their intensity. To support efficient deployment, we distill IMPACT into TIDE, a small language model that predicts contradiction evidence and intensity in a single forward pass. Experimental results show that IMPACT substantially outperforms strong single-agent and generic multi-agent baselines in both evidence identification and intensity agreement, while TIDE achieves competitive performance at significantly lower inference cost.

Related papers

• CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review [20.449658717520876]
  alg is an end-to-end multi-agent simulacra framework for studying reviewer assignment integrity.<n>We develop an affinity-guided collusive bidding strategy that uses mutual reviewer-paper affinities to construct collusion rings.
  arXiv  Detail & Related papers  (2026-09-04T14:55:33Z)
• Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes [37.743333035442426]
  This paper introduces AD2-Bench, which decomposes reasoning into a structured Chain of Evidence (CoE)<n>Building on this perspective, the authors identify two primary causes of reasoning failure: Spatial Ambiguity and Semantic Uncertainty.<n>They then propose Evidence-grounded Visual Reasoning (EGVOR), which replaces implicit reasoning with the explicit generation of Evidence Atoms.
  arXiv  Detail & Related papers  (2026-08-11T14:23:07Z)
• CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition [2.587219755021037]
  graph-based conflict-aware evidence negotiation framework for weakly supervised multimodal emotion recognition.<n>ConFER represents each modality expert as a node with a predictive belief, boundary-based uncertainty, and runtime reliability estimated from historical out-of-fold performance and current-sample uncertainty.<n>ConFER achieves competitive performance, reaching textbf0.873 accuracy on AMIGOS-V and textbf0.854 accuracy on MAHNOB-V under strict LOSO evaluation.
  arXiv  Detail & Related papers  (2026-08-08T02:42:00Z)
• ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence [48.938290419832335]
  We introduce ConflictScore, a novel metric that quantifies how well a model's response acknowledges conflicting evidence in its grounding documents.<n>Our framework decomposes responses into atomic claims, labels each claim against each grounding document, and then aggregates these labels into two complementary measures: ConflictScore-Count (CS-C), the proportion of claims exhibiting conflicts, and ConflictScore-Ratio (CS-R), the balance between supporting and contradicting evidence.
  arXiv  Detail & Related papers  (2026-06-24T23:00:09Z)
• Position: Anthropomorphic Misalignment Research Needs Stronger Evidence [52.76826423649968]
  We show how conceptual ambiguity, non-robust datasets, experimental design, and insufficient causal interventions can lead to overinterpretation of model behaviors.<n>This position paper aims to offer guidance on evidentiary considerations that can help improve methodological rigor in AMR.
  arXiv  Detail & Related papers  (2026-05-29T16:38:53Z)
• Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis [57.78407973423517]
  We advocate uncertainty-aware subjectivity analysis, where models are expected to make predictions while expressing uncertainty that reflects human disagreement.<n>To operationalize this perspective, we propose a two-phase Disagreement Perception and Uncertainty Alignment framework.<n> Experiments on three subjectivity analysis tasks show that DPUA preserves task performance while better aligning model uncertainty with human disagreement.
  arXiv  Detail & Related papers  (2026-05-11T11:52:58Z)
• To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition [87.22988227382329]
  Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict.<n>We propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities.
  arXiv  Detail & Related papers  (2026-05-06T13:11:33Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Models [99.26398772227684]
  We propose Co-FactChecker, a framework for human-AI collaborative claim verification.<n>Co-FactChecker translates expert feedback into trace-edits that introduce targeted modifications to the trace.<n>Human evaluations show that Co-FactChecker is preferred over multi-turn dialogue.
  arXiv  Detail & Related papers  (2026-04-15T10:35:00Z)
• When Contextual Inference Fails: Cancelability in Interactive Instruction Following [51.2195840589474]
  We introduce Build What I Mean, an interactive benchmark for contextual meaning construction.<n>In BWIM, models must resolve ambiguity by either performing a contextual inference or requesting clarification at a small communication cost.<n>We observe suboptimal strategies, such as partner-blind over-clarification and question-averse guessing under uncertainty.
  arXiv  Detail & Related papers  (2026-03-20T14:46:59Z)
• Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP [25.097081181685613]
  Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis.<n>We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors.<n>We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure.
  arXiv  Detail & Related papers  (2026-01-14T01:26:29Z)
• Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models [72.4149653187766]
  We propose a Reasoner-Verifier framework named Adrialversa Reasoning RAG (ARR)<n>The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage.<n> Experiments on multiple benchmarks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2026-01-08T06:57:03Z)
• Exploring Health Misinformation Detection with Multi-Agent Debate [0.11470070927586014]
  We propose a two-stage framework for health misinformation detection.<n>In the first stage, we employ large language models (LLMs) to independently evaluate retrieved articles.<n>When this score indicates insufficient consensus-falling below a predefined threshold-the system proceeds to a second stage.<n>Multiple agents engage in structured debate to synthesize conflicting evidence and generate well-reasoned verdicts with explicit justifications.
  arXiv  Detail & Related papers  (2025-11-29T12:39:30Z)
• LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents [0.10260880679794955]
  We present a multi-agent contradiction-aware benchmark framework for the legal domain.<n>It generates synthetic legal-style documents, injects six structured contradiction types, and models both self- and pairwise inconsistencies.<n>This benchmark offers one of the first structured resources for contradiction-aware evaluation in legal RAG pipelines.
  arXiv  Detail & Related papers  (2025-10-03T18:24:27Z)
• Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation [0.0]
  We present Debate, Deliberate, Decide (D3), a cost-aware, adversarial multi-agent framework that orchestrates structured debate among role-specialized agents.<n>We develop a probabilistic model of score gaps that characterizes reliability and convergence under iterative debate.<n>We show state-of-the-art agreement with human judgments, reduced positional and verbosity biases via anonymization, and a favorable cost-accuracy frontier.
  arXiv  Detail & Related papers  (2024-10-07T00:22:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.