Sibyl-AutoResearch: Autonomous Research Needs Self-Evolving Trial-and-Error Harnesses, Not Paper Generators

• URL: http://arxiv.org/abs/2605.22343v1
• Date: Thu, 21 May 2026 11:29:08 GMT
• Title: Sibyl-AutoResearch: Autonomous Research Needs Self-Evolving Trial-and-Error Harnesses, Not Paper Generators
• Abstract summary: We introduce Sibyl-AutoResearch, a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses.<n>A harness lets agents run bounded trials, preserve positive and negative outcomes, and route lessons into later planning, validation, claim scope, scheduling, critique, writing, and repair.<n>We implement the framework in SIBYL, a file-backed autonomous research system that exposes the state, roles, memory, gates, and artifact traces needed to inspect these conversion paths.
• Score: 37.075000666622074
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous research systems increasingly make the scientific workflow executable: agents can propose ideas, run code, inspect results, and draft papers. But executable workflows do not by themselves produce research judgment. We analyze where current systems lose trial experience: weak evidence becomes prose, pilot signals become broad claims, memory remains textual, and recurring process failures do not change later behavior. We introduce Sibyl-AutoResearch, a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses. A harness lets agents run bounded trials, preserve positive and negative outcomes, and route lessons into later planning, validation, claim scope, scheduling, critique, writing, and harness repair. We formalize this through two auditable conversion units: trial-to-behavior conversion, which links trial signals to later research actions, and trial-to-harness-behavior conversion, which links recurring process failures to system updates. We implement the framework in SIBYL, a file-backed autonomous research system that exposes the state, roles, memory, gates, and artifact traces needed to inspect these conversion paths. A retrospective audit identifies eight high-confidence conversion events, with a median latency of one iteration and a maximum latency of three iterations. A recovered-failure registry further shows how five naturally occurring failure classes, including duplicate results, stale numbers, and unsupported statistics, were blocked, downgraded, or routed into later repair. These traces do not establish a comparative performance claim; they show that the proposed conversion units are recoverable from realistic autonomous-research workspaces. The SIBYL framework and system are available at https://github.com/Sibyl-Research-Team/AutoResearch-SibylSystem.

Related papers

• AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems [7.383658399787325]
  In textbfDASHEN, NetEase's gaming-community app, engineers identify promising directions from research papers, technical reports, and prior production experiments.<n> Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review.<n>We present textbfAutoLR, initially built as textbfAuto Launch Review and later extended upstream into an autonomous research-to-launch harness.
  arXiv  Detail & Related papers  (2026-09-04T08:30:36Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• AgentX: Towards Agent-Driven Self-Iteration of Industrial Recommender Systems [82.01232437066487]
  AgentX is a production-deployed multi-agent system that restructures this production function.<n>It autonomously generates, implements, evaluates, and learns from recommendation experiments.<n>AgentX orchestrates four tightly coupled stages in a closed loop.
  arXiv  Detail & Related papers  (2026-06-25T10:42:28Z)
• AuditWeave: A Tamper-Evident, Auditor-Navigable Evidence Layer for AI-Assisted and Data-Transformation Workflows [0.0]
  AuditWeave is a lightweight Python library that records the steps of AI-assisted and data-transformation into a single hash-chained ledger.<n>We describe the design and evaluate recording overhead, scalability, and tamper-detection correctness on the reference implementation.
  arXiv  Detail & Related papers  (2026-06-14T21:46:04Z)
• AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery [184.2309450554096]
  AutoResearch is defined as the developmental spectrum of AI-powered scientific workflow automation.<n>We analyze how research systems redistribute control, evidence, execution, validation, and accountability across and organize the field.<n>We propose five evaluation dimensions--novelty, impact, reliability, and provenance--and show that AutoResearch autonomy is domain-conditioned.
  arXiv  Detail & Related papers  (2026-05-22T03:40:30Z)
• AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration [175.32768568927688]
  We present AutoResearchClaw, a multi-agent autonomous research pipeline built on five mechanisms.<n>On ARC-Bench, a 25-topic experiment-stage benchmark, AutoResearchClaw outperforms AI Scientist v2 by 54.7%.
  arXiv  Detail & Related papers  (2026-05-19T15:49:51Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration [10.076972559007116]
  ARIS (Auto-Research-in-sleep) is an open-source research harness for autonomous research.<n>ARIS coordinates machine-learning research through cross-model adversarial collaboration.
  arXiv  Detail & Related papers  (2026-05-04T18:10:15Z)
• AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation [8.807417226975081]
  AutoResearch is an execution-grounded multi-agent framework for reliable research workflow automation.<n>System treats errors, citation-verification failures, and review-agent feedback as practical filtering signals for generated research artifacts.
  arXiv  Detail & Related papers  (2026-05-04T08:24:21Z)
• RESCORE: LLM-Driven Simulation Recovery in Control Systems Research Papers [18.24843021913362]
  Reconstructing numerical simulations from control systems research papers is often hindered by underspecified parameters and ambiguous implementation details.<n>We define the task of Paper to Simulation Recoverability, the ability of an automated system to generate executable code that faithfully reproduces a paper's results.<n>We propose RESCORE, a three component LLM agentic framework, Analyzer, Coder, and Verifier.<n>Our method successfully recovers task coherent simulations for 40.7% of benchmark instances, outperforming single pass generation.
  arXiv  Detail & Related papers  (2026-04-06T00:13:14Z)
• AI-Assisted Unit Test Writing and Test-Driven Code Refactoring: A Case Study [0.0]
  We present a case study of using coding models for automated unit test generation and subsequent safe iteration.<n>We generated nearly 16,000 lines of reliable unit tests in hours rather than weeks, achieved up to 78% branch coverage in critical modules, and significantly reduced regression risk during large-scale iterations.
  arXiv  Detail & Related papers  (2026-04-03T15:54:43Z)
• Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate [8.424102114588559]
  Automated Program Repair (APR) struggles with complex logic errors and silent failures.<n>Current LLM-based APR methods are mostly static, relying on source code and basic test outputs.<n>We propose TraceRepair, a multi-agent framework that leverages runtime facts as shared constraints for patch validation.
  arXiv  Detail & Related papers  (2026-04-03T02:23:25Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches [0.3637090866781549]
  Reproducing computational research is often assumed to be as simple as rerunning the original code with provided data.<n>This study investigates whether large language models and AI agents can automate the diagnosis and repair of such failures.
  arXiv  Detail & Related papers  (2026-02-09T11:59:59Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.