Review the Code, Not the Story: A Vision and Protocol for Code-First Peer Review

• URL: http://arxiv.org/abs/2606.07683v1
• Date: Fri, 05 Jun 2026 01:26:07 GMT
• Title: Review the Code, Not the Story: A Vision and Protocol for Code-First Peer Review
• Abstract summary: This vision and protocol paper proposes code-first peer review.<n>Authors submit executable research artifacts and minimal claim manifests.<n>A venue-controlled AI system builds the environment, executes experiments, audits code paths, maps claims to evidence, and generates a standardized Review Package for human reviewers.
• Score: 2.531440973296736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Peer review in computational fields remains centered on author-written manuscripts, even though the decisive evidence for many claims resides in executable code, data, configurations, and experiment pipelines. This manuscript-first workflow gives authors substantial control over narrative framing while leaving reviewers with limited time to inspect implementation details, reproduce results, or detect unsupported claims. This vision and protocol paper proposes code-first peer review: authors submit executable research artifacts and minimal claim manifests; a venue-controlled AI system builds the environment, executes experiments, audits code paths, maps claims to evidence, and generates a standardized Review Package for human reviewers. The goal is not to replace reviewers or to give authors an automatic writing assistant. Instead, AI serves as review infrastructure that shifts the target of peer review from polished narratives to executable evidence. We formalize a claim-evidence contract, define the Generated Review View and Review Package abstractions, give a worked example, outline a system architecture, and analyze evaluation and governance challenges including AI bias, prompt injection, model instability, auditability, and author appeal.

Related papers

• ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation [54.82704239643649]
  We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans.<n>Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback.<n>We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.
  arXiv  Detail & Related papers  (2026-09-08T17:30:16Z)
• ResearchLoop: An Evidence-Gated Control Plane for AI-Assisted Research [4.9833735627186435]
  ResearchLoop is an evidence-gated control plane for AI-assisted computational research.<n>ResearchLoop treats research questions, task contracts, evidence objects, claim ledgers, closeouts, and paper bindings as durable project state.<n>This technical report provides the complete protocol specification, state model, transition rules, claim-admission algorithm, and insight-compounding mechanism.
  arXiv  Detail & Related papers  (2026-05-27T10:29:00Z)
• Making AI-Assisted Grant Evaluation Auditable without Exposing the Model [0.0]
  We propose a TEE-based architecture that helps reconcile requirements through remote attestation.<n>The architecture allows an external verifier to check which model, rubric, prompt template, and input representation were used.<n>We position the design relative to confidential AI inference, attestable AI audits, zero-knowledge machine learning, algorithmic accountability, and AI-assisted peer review.
  arXiv  Detail & Related papers  (2026-04-28T04:10:04Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Peerispect: Claim Verification in Scientific Peer Reviews [12.340116689292726]
  Peerispect is an interactive system that operationalizes claim-level verification in peer reviews.<n>Results are presented through a visual interface that highlights evidence directly in the paper.
  arXiv  Detail & Related papers  (2026-04-19T23:40:26Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review [53.99984738447279]
  Recent work frames this task as automatic text generation, underusing author expertise and intent.<n>We introduce REspGen, a generation framework that integrates explicit author input, multi-attribute control, and evaluation-guided refinement.<n>To support this formulation, we construct Re$3$Align, the first large-scale dataset of aligned review-response--revision triplets.
  arXiv  Detail & Related papers  (2026-01-19T14:07:10Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Generative Adversarial Reviews: When LLMs Become the Critic [1.2430809884830318]
  We introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers.<n>Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information.<n>Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes.
  arXiv  Detail & Related papers  (2024-12-09T06:58:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.