How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
- URL: http://arxiv.org/abs/2608.14905v2
- Date: Wed, 19 Aug 2026 01:11:52 GMT
- Title: How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
- Abstract summary: AutoResearchEval features 100 tasks grounded in published frontier science across 7 scientific domains.<n> evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories.<n>We leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts.
- Score: 6.001213285107465
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery.
Related papers
- SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents [56.400332595452966]
We introduce SearchAuditBench, a benchmark that evaluates whether auditors can localize, attribute, and repair search-agent failures.<n>SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks.<n>We propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures.
arXiv Detail & Related papers (2026-08-05T09:09:00Z) - AREX: Towards a Recursively Self-Improving Agent for Deep Research [75.96468303743958]
We introduce AREX, a family of Recursively Self-Improving deep research agents.<n>AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer.<n>We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning.
arXiv Detail & Related papers (2026-07-23T16:05:46Z) - FARS: A Fully Automated Research System Deployed at Scale [19.413175066072302]
We present FARS (Fully Automated Research System), a fully automated AI-for-AI research system.<n> FARS autonomously generates and advances projects through ideation, planning, experimentation, and writing.<n>In its first public deployment, FARS produced 166 complete research papers spanning 67 fine-grained AI/ML topics.
arXiv Detail & Related papers (2026-06-30T13:30:24Z) - Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
arXiv Detail & Related papers (2026-06-10T22:55:30Z) - Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle [16.040691949938203]
We propose the AARR (Act As a Real Researcher) benchmark series.<n>AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers.<n>Our results indicate that developing researcher-like AI requires further exploration of research behavior.
arXiv Detail & Related papers (2026-06-05T17:13:36Z) - AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration [175.74514061083195]
We present AutoResearchClaw, a multi-agent autonomous research pipeline built on five mechanisms.<n>On ARC-Bench, a 25-topic experiment-stage benchmark, AutoResearchClaw outperforms AI Scientist v2 by 54.7%.
arXiv Detail & Related papers (2026-05-19T15:49:51Z) - AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
arXiv Detail & Related papers (2026-05-18T17:08:26Z) - The Last Human-Written Paper: Agent-Native Research Artifacts [106.47848184955576]
We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package.<n>Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste.
arXiv Detail & Related papers (2026-04-27T16:23:09Z) - AI-Supervisor: Autonomous AI Research Supervision via a Persistent Research World Model [1.14219428942199]
Existing automated research systems operate as stateless, linear pipelines.<n>We present textbfAI-Supervisor, a multi-agent orchestration framework.<n>Agents provide end-to-end AI research supervision driven by human interests.
arXiv Detail & Related papers (2026-03-25T15:16:51Z) - FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
arXiv Detail & Related papers (2026-02-02T23:21:13Z) - When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research [19.97666809905332]
Large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.<n>Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.
arXiv Detail & Related papers (2025-05-17T05:45:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.