Training AI Scientists to Replicate Research

• URL: http://arxiv.org/abs/2608.13331v1
• Date: Thu, 13 Aug 2026 14:59:27 GMT
• Title: Training AI Scientists to Replicate Research
• Abstract summary: We develop Replica, a scalable task space for paper replication.<n>We introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality.
• Score: 8.913339899077362
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we develop Replica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-train Faraday, a 27B-parameter "AI Scientist" agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

Related papers

• SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning [54.194672921128785]
  We introduce SciResearcher, a fully automated agentic framework for frontier-science data construction.<n>SciResearcher synthesizes diverse conceptual and computational tasks grounded in academic evidence.<n>We develop SciResearcher-8B, an agent foundation model that achieves 19.46% on the HLE-Bio/Chem-Gold benchmark.
  arXiv  Detail & Related papers  (2026-05-02T15:26:45Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• OR-Agent: Bridging Evolutionary Search and Structured Research for Automated Algorithm Discovery [10.217363774023033]
  OR-Agent is a multi-agent research framework designed for automated exploration in rich experimental environments.<n>We introduce an evolutionary-systematic mechanism that unifies evolutionary selection of research starting points, comprehensive research plan generation, and coordinated exploration within a research tree.<n>We conduct experiments across classical optimization benchmarks-including traveling salesman, capacitated vehicle routing, bin packing, orienteering, and multiple knapsack problems-as well as a simulation-based cooperative driving scenarios.
  arXiv  Detail & Related papers  (2026-02-14T13:32:03Z)
• ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences [19.81372090301296]
  ReplicatorBench is an end-to-end benchmark for evaluating AI agents in research replication across three stages.<n>We develop ReplicatorAgent, an agentic framework equipped with necessary tools, like web search and iterative interaction with sandboxed environments.<n>We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access.
  arXiv  Detail & Related papers  (2026-02-11T20:42:10Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR [64.22412492998754]
  We release a search corpus of 16 million biomedical paper abstracts and construct a challenging factoid QA dataset called PaperSearchQA.<n>We train search agents in this environment to outperform non-RL retrieval baselines.<n>Our data creation methods are scalable and easily extendable to other scientific domains.
  arXiv  Detail & Related papers  (2026-01-26T06:46:16Z)
• ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? [29.17900668495058]
  We introduce ReplicationBench, an evaluation framework for frontier AI agents.<n>It tests whether agents can replicate entire research papers drawn from the astrophysics literature.<n>R ReplicationBench establishes the first benchmark of paper-scale, expert-validated astrophysics research tasks.
  arXiv  Detail & Related papers  (2025-10-28T16:21:19Z)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite [75.58737079136942]
  We present AstaBench, a suite that provides the first holistic measure of agentic ability to perform scientific research.<n>Our suite comes with the first scientific research environment with production-grade search tools.<n>Our evaluation of 57 agents across 22 agent classes reveals several interesting findings.
  arXiv  Detail & Related papers  (2025-10-24T17:10:26Z)
• Operationalizing Serendipity: Multi-Agent AI Workflows for Enhanced Materials Characterization with Theory-in-the-Loop [0.0]
  SciLink is an open-source, multi-agent artificial intelligence framework designed to operationalize serendipity in materials research.<n>It creates a direct, automated link between experimental observation, novelty assessment, and theoretical simulations.<n>We show its application to atomic-resolution and hyperspectral data, its capacity to integrate real-time human expert guidance, and its ability to close the research loop.
  arXiv  Detail & Related papers  (2025-08-07T04:59:17Z)
• AI Copilots for Reproducibility in Science: A Case Study [1.583709163934932]
  Open science initiatives seek to make research outputs more transparent, accessible, and reusable, but ensuring that published findings can be independently reproduced remains a persistent challenge.<n>This paper introduces OpenPub, an AI-powered platform that supports researchers, reviewers, and readers through a suite of modular copilots focused on key open science tasks.
  arXiv  Detail & Related papers  (2025-06-25T04:56:28Z)
• ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition [67.26124739345332]
  Large language models (LLMs) have demonstrated potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined.<n>We introduce the first large-scale benchmark for evaluating LLMs with a near-sufficient set of sub-tasks of scientific discovery.<n>We develop an automated framework that extracts critical components - research questions, background surveys, inspirations, and hypotheses - from scientific papers.
  arXiv  Detail & Related papers  (2025-03-27T08:09:15Z)
• CycleResearcher: Improving Automated Research via Automated Review [37.03497673861402]
  This paper explores the possibility of using open-source post-trained large language models (LLMs) as autonomous agents capable of performing the full cycle of automated research and review.<n>To train these models, we develop two new datasets, reflecting real-world machine learning research and peer review dynamics.<n>Our results demonstrate that CycleReviewer achieves promising performance with a 26.89% reduction in mean absolute error (MAE) compared to individual human reviewers in predicting paper scores.
  arXiv  Detail & Related papers  (2024-10-28T08:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.