Toward Autonomous Long-Horizon Engineering for ML Research

• URL: http://arxiv.org/abs/2604.13018v1
• Date: Tue, 14 Apr 2026 17:55:16 GMT
• Title: Toward Autonomous Long-Horizon Engineering for ML Research
• Abstract summary: AiScientist is a system for autonomous long-horizon engineering for ML research built on a simple principle.<n>AiScientist combines hierarchical orchestration with a permission-scoped File-as-Bus workspace.
• Score: 89.63005598358639
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous AI research has advanced rapidly, but long-horizon ML research engineering remains difficult: agents must sustain coherent progress across task comprehension, environment setup, implementation, experimentation, and debugging over hours or days. We introduce AiScientist, a system for autonomous long-horizon engineering for ML research built on a simple principle: strong long-horizon performance requires both structured orchestration and durable state continuity. To this end, AiScientist combines hierarchical orchestration with a permission-scoped File-as-Bus workspace: a top-level Orchestrator maintains stage-level control through concise summaries and a workspace map, while specialized agents repeatedly re-ground on durable artifacts such as analyses, plans, code, and experimental evidence rather than relying primarily on conversational handoffs, yielding thin control over thick state. Across two complementary benchmarks, AiScientist improves PaperBench score by 10.54 points on average over the best matched baseline and achieves 81.82 Any Medal% on MLE-Bench Lite. Ablation studies further show that File-as-Bus protocol is a key driver of performance, reducing PaperBench by 6.41 points and MLE-Bench Lite by 31.82 points when removed. These results suggest that long-horizon ML research engineering is a systems problem of coordinating specialized work over durable project state, rather than a purely local reasoning problem.

Related papers

• Toward Generalist Autonomous Research via Hypothesis-Tree Refinement [150.99641031769633]
  We introduce Arbor, a framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR)<n>As results return, Arbor updates the tree, propagates reusable lessons, refines the search frontier, and admits verified improvements.
  arXiv  Detail & Related papers  (2026-06-10T10:57:05Z)
• AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? [84.49732367417067]
  We introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization.<n>AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains.<n>We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.
  arXiv  Detail & Related papers  (2026-06-03T16:36:54Z)
• When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution [17.020655152761346]
  LongAct is a benchmark designed to evaluate planning-level autonomy in long-horizon household tasks.<n>HoloMind is a VLM-driven agent with a DAG-based long-horizon hierarchical planner.
  arXiv  Detail & Related papers  (2026-05-14T07:47:53Z)
• The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break [69.91172974854422]
  HORIZON is a cross-domain diagnostic benchmark for analyzing long-horizon failure behaviors in large language model (LLM) agents.<n>We evaluate state-of-the-art (SOTA) agents from multiple model families to study horizon-dependent degradation patterns.<n>Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures.
  arXiv  Detail & Related papers  (2026-04-13T19:11:42Z)
• LUMINA: LLM-Guided GPU Architecture Exploration via Bottleneck Analysis [8.471843667828994]
  Large-Language Model (LLM) inference is challenging because of GPU's vast, multi-modal design spaces.<n>Existing automated DSE methods are often prohibitively expensive.<n>We present LUMINA, an LLM-driven GPU architecture exploration framework that leverage AI to enhance the DSE efficiency.
  arXiv  Detail & Related papers  (2026-03-06T04:47:18Z)
• LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces [65.11019654023978]
  LongCLI-Bench is a benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks.<n>We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world tasks.<n>Experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench.
  arXiv  Detail & Related papers  (2026-02-15T23:12:57Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• Toward Ultra-Long-Horizon Agentic Science: Cognitive Accumulation for Machine Learning Engineering [59.18634614089481]
  We present ML-Master 2.0, an autonomous agent that masters ultra-long-horizon machine learning engineering (MLE)<n>By reframing context management as a process of cognitive accumulation, our approach introduces Hierarchical Cognitive Caching (HCC)<n>HCC allows agents to decouple immediate execution from long-term experimental strategy.<n>In evaluations on OpenAI's MLE-Bench under 24-hour budgets, ML-Master 2.0 achieves a state-of-the-art medal rate of 56.44%.
  arXiv  Detail & Related papers  (2026-01-15T13:52:04Z)
• OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows [10.318744035680398]
  Large language models (LLMs) are increasingly deployed in real-world applications requiring complex, long-horizon reasoning.<n>OdysseyBench is a comprehensive benchmark for evaluating LLM agents on long-horizon across diverse office applications.<n>To enable scalable benchmark creation, we propose HomerAgents, a multi-agent framework that automates the generation of long-horizon workflow benchmarks.
  arXiv  Detail & Related papers  (2025-08-12T17:53:03Z)
• Beyond Needle(s) in the Embodied Haystack: Environment, Architecture, and Training Considerations for Long Context Reasoning [11.179019629415514]
  $infty$-THOR is a new framework for long-horizon embodied tasks that advances long-context understanding in embodied AI.<n>$infty$-THOR provides: (1) a generation framework for scalable, reproducible, and unlimited long-horizon trajectories; (2) a novel embodied QA task, Needle(s) in the Embodied Haystack; and (3) a long-horizon dataset and benchmark suite.
  arXiv  Detail & Related papers  (2025-05-22T17:20:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.