Coding-agents can replicate scientific machine learning papers

• URL: http://arxiv.org/abs/2607.02134v1
• Date: Thu, 02 Jul 2026 13:11:30 GMT
• Title: Coding-agents can replicate scientific machine learning papers
• Abstract summary: Paper-replication makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill.<n>The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims.<n>We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.

Related papers

• Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill [8.917367808177904]
  We present Spark-to-Paper, an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant.<n>It separates experiment planning from reporting, so that required evidence is specified before results are observed and manuscript claims are revised according to measured outcomes.<n>Across eight controlled research topics, Spark-to-Paper achieves 99.5% citation validity and 96.4% figure editability.
  arXiv  Detail & Related papers  (2026-08-12T11:11:07Z)
• DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments [51.049999642138]
  We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence.<n>Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly.<n>Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops.
  arXiv  Detail & Related papers  (2026-07-19T15:20:40Z)
• ResearchLoop: An Evidence-Gated Control Plane for AI-Assisted Research [4.9833735627186435]
  ResearchLoop is an evidence-gated control plane for AI-assisted computational research.<n>ResearchLoop treats research questions, task contracts, evidence objects, claim ledgers, closeouts, and paper bindings as durable project state.<n>This technical report provides the complete protocol specification, state model, transition rules, claim-admission algorithm, and insight-compounding mechanism.
  arXiv  Detail & Related papers  (2026-05-27T10:29:00Z)
• Rollout Cards: A Reproducibility Standard for Agent Research [15.381365113892848]
  Papers compare systems by reported scores while leaving rollout records behind those scores difficult to inspect.<n>For agentic tasks, this matters because the same behaviour can receive different reported scores when evaluations select different parts of a rollout or apply different reporting rules.<n>In a structured audit of 50 popular training and evaluation repositories, we find that none report how many runs failed, errored, or were skipped alongside headline scores.
  arXiv  Detail & Related papers  (2026-05-12T13:54:31Z)
• AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation [8.807417226975081]
  AutoResearch is an execution-grounded multi-agent framework for reliable research workflow automation.<n>System treats errors, citation-verification failures, and review-agent feedback as practical filtering signals for generated research artifacts.
  arXiv  Detail & Related papers  (2026-05-04T08:24:21Z)
• Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results [46.81245843451412]
  We develop an agentic reproduction system that extracts structured methods descriptions from papers.<n>Agents never see the original code, results, or paper.<n>An error attribution step traces through the system chain to identify root causes.
  arXiv  Detail & Related papers  (2026-04-23T17:59:18Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction [57.86097956633207]
  method is a graph-based agent framework for generating executable code from academic papers.<n>On an extended ReproduceBench spanning 3 domains, 10 tasks, and 40 recent papers, method achieves an average performance gap of 10.04% against official implementations.
  arXiv  Detail & Related papers  (2026-03-02T12:33:31Z)
• Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents [8.185402940269794]
  We propose a prompt-free collaborative agent framework that automatically enhances the quality of paper-to-code generation.<n>Our approach employs two collaborative agents: a verification agent that examines whether the outputs at each step satisfy the requirements specified in the corresponding system prompt, and a refinement agent that revises the outputs based on the identified issues.
  arXiv  Detail & Related papers  (2025-12-02T14:24:23Z)
• Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification [46.845133190560375]
  Motivated by how humans use systematic checklists to efficiently debug complex code, we propose textbfRePro, a textbfReflective Paper-to-Code textbfReproduction framework.<n>It automatically extracts a paper's fingerprint, referring to a comprehensive set of accurate and atomic criteria serving as high-quality supervisory signals.<n>It achieves 13.0% performance gap over baselines, and it correctly revises complex logical and mathematical criteria in reflecting.
  arXiv  Detail & Related papers  (2025-08-21T06:57:44Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.