ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

• URL: http://arxiv.org/abs/2605.26340v1
• Date: Mon, 25 May 2026 21:30:27 GMT
• Title: ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence
• Abstract summary: Chain-of-Evidence (CoE) is a verifiability framework requiring every claim to be traceable to its evidence source.<n>CoE Audit is a post-hoc audit whose four integrity checks -- score verification, specification violation, reference verification, and method-code alignment -- apply uniformly to all systems.
• Score: 57.37494162084001
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous research agents produce competitive solutions and professional-looking manuscripts, yet their outputs contain verifiability failures undetectable by surface-level evaluation: fabricated citations, unreproducible scores, and method descriptions that diverge from the implementation. We address this through three contributions. First, Chain-of-Evidence (CoE), a verifiability framework requiring every claim to be traceable to its evidence source. Second, ScientistOne, an end-to-end autonomous research system that maintains evidence chains by construction throughout literature review, solution discovery, and paper writing. Third, CoE Audit, a post-hoc audit whose four integrity checks -- score verification, specification violation, reference verification, and method-code alignment -- apply uniformly to all systems. Across 75 papers spanning five systems and five frontier research tasks, every baseline exhibits at least one systematic failure mode: hallucinated reference rates reach 21%, score verification passes in as few as 42% of papers, and method-code alignment ranges from 20% to 80%. ScientistOne achieves zero hallucinated references (0/337), perfect score verification (12/12), and the highest method-code alignment (14/15), while matching or exceeding human expert performance on all five tasks. ScientistOne further generalizes to six additional tasks spanning medical imaging, fine-grained recognition, 3D perception, and language modeling, achieving state-of-the-art on Parameter Golf and gold medals on MLE-Bench tasks where baselines fail entirely.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents [16.43230671992556]
  AutoSciRub is an evaluation-first framework that induces a task-specific executable before research execution.<n>AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria.<n>During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report.
  arXiv  Detail & Related papers  (2026-08-31T16:48:51Z)
• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• OmniScientist: An Omni-Modal Omni-Discipline AI Scientist [39.38448883079899]
  We introduce OmniScientist, an end-to-end, omni-modal AI scientist that conducts research directly from raw evidence.<n>A perception layer and 3 autonomous agents for ideation, experiment, and writeup operate within a deterministic pipeline.<n>We evaluate OmniScientist on 36 real-data cases spanning 5 discipline families, 4 families of scientific evidence, and modalities including images, signals, audio, video, 3-D structures, trajectories, tables, formulae, and graphs.
  arXiv  Detail & Related papers  (2026-08-13T17:59:52Z)
• Knowledge Synthesis Review Framework: Task-Level Benchmarking of LLM-Based Systems for Multi-Source Evidence Synthesis [0.0]
  Large language models (LLMs) may accelerate this work, but their reliability across cognitive tasks of a review remains uncertain.<n>We introduce the Knowledge Synthesis Review (KSR), a human-in-the-loop framework that decomposes evidence synthesis into screening, extraction, analysis, and synthesis.<n>We evaluated GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, and NotebookLM on a 244-document benchmark subset drawn from a 1,893-document corpus on AI.
  arXiv  Detail & Related papers  (2026-08-13T02:41:14Z)
• SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents [56.400332595452966]
  We introduce SearchAuditBench, a benchmark that evaluates whether auditors can localize, attribute, and repair search-agent failures.<n>SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks.<n>We propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures.
  arXiv  Detail & Related papers  (2026-08-05T09:09:00Z)
• Prompt-to-Paper: Agentic AI System for Bioinformatics [0.0]
  We present Prompt-to-Paper, a multi-agent framework that directly addresses this evaluation gap through three integrated innovations.<n>First, a deterministic retrieval-augmented generation pipeline with section-aware relevance scoring and snowball citation expansion grounds every claim in a verifiable corpus of 60--100 papers.<n>Second, an autonomous coding agent executes real computational biology experiments replacing synthetic outputs with genuine numerical results.<n>Third, an eight-dimensional automated quality scorer, benchmarked with approximate reference statistics from published papers and augmented with explicit hallucination penalties, provides standardized, reproducible quality assessments.
  arXiv  Detail & Related papers  (2026-07-05T21:30:24Z)
• Deterministic Integrity Gates for LLM-Assisted Clinical Manuscript Preparation: An Auditable Biomedical Informatics Architecture [0.3823356975862005]
  We describe an architecture pairing generation with verification, resting on three principles.<n>This determinism-where-possible split, organized as an integrity-gate taxonomy, is the core contribution.<n> MedSci Skills is an open-source toolkit of 43 skills with a 21-detector deterministic tier, evaluated on three public-dataset pipelines.
  arXiv  Detail & Related papers  (2026-06-08T13:51:04Z)
• Inductive Deductive Synthesis: Enabling AI to Generate Formally Verified Systems [100.24694338574402]
  We present Inductive Deductive Synthesis (IDS), which jointly and incrementally synthesizes implementation and proof.<n>IDS achieves 7/7 in about 6.8 hours and $106 per spec on average, roughly 200x faster than expert effort and 17% cheaper than SOTA agents.<n>IDS further incorporates performance feedback into the same loop, yielding implementations up to 3x faster than published verified systems.
  arXiv  Detail & Related papers  (2026-05-22T00:05:36Z)
• The Last Human-Written Paper: Agent-Native Research Artifacts [106.47848184955576]
  We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package.<n>Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste.
  arXiv  Detail & Related papers  (2026-04-27T16:23:09Z)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome [109.15093810810214]
  MiroEval is a benchmark and evaluation framework for deep research systems.<n>The benchmark comprises 100 tasks grounded in real user needs.<n>The proposed evaluation suite assesses deep research systems along three complementary dimensions.
  arXiv  Detail & Related papers  (2026-03-30T13:16:03Z)
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment [63.662126457336534]
  OpenNovelty is an agentic system for transparent, evidence-based novelty analysis.<n>It grounds all assessments in retrieved real papers, ensuring verifiable judgments.<n>OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
  arXiv  Detail & Related papers  (2026-01-04T15:48:51Z)
• AI-Powered Citation Auditing: A Zero-Assumption Protocol for Systematic Reference Verification in Academic Research [0.0]
  This paper presents a novel AI-powered methodology for systematic, comprehensive reference auditing using agentic AI with tool-use capabilities.<n>We develop a zero-assumption verification protocol that independently validates every reference against multiple academic databases without assuming any citation is correct.<n>Results demonstrate 91.7% average verification rate on published PLOS papers, with successful detection of fabricated references, retracted articles, orphan citations, and predatory journals.<n>This work establishes the first validated AI-agent methodology for academic citation integrity, demonstrating practical applicability for supervisors, students, and institutional quality assurance.
  arXiv  Detail & Related papers  (2025-10-17T16:53:03Z)
• SurveyBench: Can LLM(-Agents) Write Academic Surveys that Align with Reader Needs? [37.28508850738341]
  Survey writing is a labor-intensive and intellectually demanding task.<n>Recent approaches, such as general DeepResearch agents and survey-specialized methods, can generate surveys automatically.<n>But their outputs often fall short of human standards and there lacks a rigorous, reader-aligned benchmark.<n>We propose a fine-grained, quiz-driven evaluation framework SurveyBench.
  arXiv  Detail & Related papers  (2025-10-03T15:49:09Z)
• HySemRAG: A Hybrid Semantic Retrieval-Augmented Generation Framework for Automated Literature Synthesis and Methodological Gap Analysis [55.2480439325792]
  HySemRAG is a framework that combines Extract, Transform, Load (ETL) pipelines with Retrieval-Augmented Generation (RAG)<n>System addresses limitations in existing RAG architectures through a multi-layered approach.
  arXiv  Detail & Related papers  (2025-08-01T20:30:42Z)
• SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation [2.985620880452744]
  SciSage is a multi-agent framework employing a reflect-when-you-write paradigm.<n>It critically evaluates drafts at outline, section, and document levels, collaborating with specialized agents for query interpretation, content retrieval, and refinement.<n>We also release SurveyScope, a benchmark of 46 high-impact papers ( 2020-2025) across 11 computer science domains.
  arXiv  Detail & Related papers  (2025-06-15T02:23:47Z)
• ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery [23.773528748933934]
  We present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery.<n>We extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them.<n>Using ScienceAgentBench, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands CodeAct, and self-AI.
  arXiv  Detail & Related papers  (2024-10-07T14:33:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.