MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility

• URL: http://arxiv.org/abs/2605.16616v1
• Date: Fri, 15 May 2026 20:35:32 GMT
• Title: MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility
• Abstract summary: We introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning.<n>The benchmark was constructed from ICML 2025 papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems.<n>In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and failures across all systems.
• Score: 2.2969445973129123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous research systems capable of generating complete scientific manuscripts have advanced rapidly, yet robust and realistic evaluation frameworks have failed to keep pace. To bridge this gap, we introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning reproducibility. The benchmark was constructed from ICML 2025 outstanding papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems: AI SCIENTIST-V1, AI SCIENTIST-V2, AGENT LABORATORY, CYCLERESEARCHER, AI RESEARCHER, and TINY SCIENTIST, yielding 45 generated manuscripts, with 3 failed experiments. Outputs are assessed using a dual-protocol approach that combines automated conference-style review and structured expert human evaluation, while tracking computational cost, runtime, and the amount of required human intervention. The automated conference-style review accepted 10 out of 37 valid submissions. An additional 8 submissions were desk-rejected before review for failing to meet the minimum page threshold. In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and reproducibility failures across all systems, and 59% of accepted automated reviews contained fabricated or unsupported claims. We further find that neither token budget nor computational cost predicts output quality: the cheapest system outperforms the most resource-intensive system in human evaluation, despite a 38-fold difference in input tokens. We thus demonstrate that autonomous research workflow design matters more than the scale of compute. MLReplicate exposes a substantial gap between current autonomous research systems and genuine scientific rigor, and establishes a practical, extensible evaluation framework for systematic progress toward trustworthy AI-driven scientific discovery.

Related papers

• ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent System [29.871573462927643]
  We introduce ReasFlow, an end-to-end autonomous agent system for reasoning-centric scientific discovery.<n>The system unifies literature synthesis, algorithm design, theorem proving, experimentation, and manuscript preparation in a single system.<n>ReasFlow is publicly accessible via the ReasLab platform, providing a collaborative workspace for AI-assisted theoretical research.
  arXiv  Detail & Related papers  (2026-07-15T13:27:20Z)
• AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery [184.2309450554096]
  AutoResearch is defined as the developmental spectrum of AI-powered scientific workflow automation.<n>We analyze how research systems redistribute control, evidence, execution, validation, and accountability across and organize the field.<n>We propose five evaluation dimensions--novelty, impact, reliability, and provenance--and show that AutoResearch autonomy is domain-conditioned.
  arXiv  Detail & Related papers  (2026-05-22T03:40:30Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery [49.77172027295716]
  AutoSOTA is an end-to-end automated research system that advances the latest State-Of-The-Art (SOTA) models published in top-tier AI papers.<n>It successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper.
  arXiv  Detail & Related papers  (2026-04-07T07:52:01Z)
• Towards a Medical AI Scientist [73.6056699962416]
  We introduce Medical AI Scientist, the first autonomous research framework tailored to clinical autonomous research.<n>The framework operates under 3 research modes, namely paper-based reproduction, literature-inspired innovation, and task-driven exploration.<n>Our system achieves strong alignment between the proposed method and its implementation, while also demonstrating significantly higher success rates in executable experiments.
  arXiv  Detail & Related papers  (2026-03-30T15:37:25Z)
• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• From Task Executors to Research Partners: Evaluating AI Co-Pilots Through Workflow Integration in Biomedical Research [0.16174969956296248]
  This rapid review examines benchmarking practices for AI systems in preclinical biomedical research.<n>A process-oriented evaluation framework is proposed that addresses four critical dimensions absent from current benchmarks.<n>These dimensions are essential for evaluating AI systems as research co-pilots rather than as isolated task executors.
  arXiv  Detail & Related papers  (2025-12-04T14:37:46Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research [19.97666809905332]
  Large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.<n>Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.
  arXiv  Detail & Related papers  (2025-05-17T05:45:16Z)
• CycleResearcher: Improving Automated Research via Automated Review [37.03497673861402]
  This paper explores the possibility of using open-source post-trained large language models (LLMs) as autonomous agents capable of performing the full cycle of automated research and review.<n>To train these models, we develop two new datasets, reflecting real-world machine learning research and peer review dynamics.<n>Our results demonstrate that CycleReviewer achieves promising performance with a 26.89% reduction in mean absolute error (MAE) compared to individual human reviewers in predicting paper scores.
  arXiv  Detail & Related papers  (2024-10-28T08:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.