VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

• URL: http://arxiv.org/abs/2607.02931v1
• Date: Fri, 03 Jul 2026 03:55:59 GMT
• Title: VERITAS: Towards a General-Purpose Replication Tool for Scientific Research
• Abstract summary: We present VERITAS, a domain-agnostic replication framework built around CLI coding agents.<n>Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs.<n>The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched.<n>We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics.
• Score: 19.45546798679836
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.

Related papers

• AI Coding Agents Can Reproduce Social Science Findings [5.410553210073652]
  Recent evidence suggests that AI coding agents can reproduce published findings when provided with original data and code.<n>We introduce SocSci-Repro-Bench, a benchmark of 221 tasks spanning four disciplines and 13 substantive domains.<n>We find that both can reproduce a large share of social science findings, with Claude Code substantially outperforming Codex.
  arXiv  Detail & Related papers  (2026-06-09T21:00:05Z)
• AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation [8.807417226975081]
  AutoResearch is an execution-grounded multi-agent framework for reliable research workflow automation.<n>System treats errors, citation-verification failures, and review-agent feedback as practical filtering signals for generated research artifacts.
  arXiv  Detail & Related papers  (2026-05-04T08:24:21Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework [82.66443886385125]
  Paper Circle is a discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature.<n>The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-agent scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes such as concepts, methods, and figures.
  arXiv  Detail & Related papers  (2026-04-07T17:59:58Z)
• ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences [19.81372090301296]
  ReplicatorBench is an end-to-end benchmark for evaluating AI agents in research replication across three stages.<n>We develop ReplicatorAgent, an agentic framework equipped with necessary tools, like web search and iterative interaction with sandboxed environments.<n>We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access.
  arXiv  Detail & Related papers  (2026-02-11T20:42:10Z)
• Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches [0.3637090866781549]
  Reproducing computational research is often assumed to be as simple as rerunning the original code with provided data.<n>This study investigates whether large language models and AI agents can automate the diagnosis and repair of such failures.
  arXiv  Detail & Related papers  (2026-02-09T11:59:59Z)
• Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification [46.845133190560375]
  Motivated by how humans use systematic checklists to efficiently debug complex code, we propose textbfRePro, a textbfReflective Paper-to-Code textbfReproduction framework.<n>It automatically extracts a paper's fingerprint, referring to a comprehensive set of accurate and atomic criteria serving as high-quality supervisory signals.<n>It achieves 13.0% performance gap over baselines, and it correctly revises complex logical and mathematical criteria in reflecting.
  arXiv  Detail & Related papers  (2025-08-21T06:57:44Z)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking [48.90371827091671]
  AutoExperiment is a benchmark that evaluates AI agents' ability to implement and run machine learning experiments.<n>We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases.<n>Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution.
  arXiv  Detail & Related papers  (2025-06-24T15:39:20Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [62.049868205196425]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)
• Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning [70.04746094652653]
  We introduce PaperCoder, a framework that transforms machine learning papers into functional code repositories.<n>PaperCoder operates in three stages: planning, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files.<n>We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations.
  arXiv  Detail & Related papers  (2025-04-24T01:57:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.