The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

• URL: http://arxiv.org/abs/2604.11978v1
• Date: Mon, 13 Apr 2026 19:11:42 GMT
• Title: The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
• Abstract summary: HORIZON is a cross-domain diagnostic benchmark for analyzing long-horizon failure behaviors in large language model (LLM) agents.<n>We evaluate state-of-the-art (SOTA) agents from multiple model families to study horizon-dependent degradation patterns.<n>Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures.
• Score: 69.91172974854422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents perform strongly on short- and mid-horizon tasks, but often break down on long-horizon tasks that require extended, interdependent action sequences. Despite rapid progress in agentic systems, these long-horizon failures remain poorly characterized, hindering principled diagnosis and comparison across domains. To address this gap, we introduce HORIZON, an initial cross-domain diagnostic benchmark for systematically constructing tasks and analyzing long-horizon failure behaviors in LLM-based agents. Using HORIZON, we evaluate state-of-the-art (SOTA) agents from multiple model families (GPT-5 variants and Claude models), collecting 3100+ trajectories across four representative agentic domains to study horizon-dependent degradation patterns. We further propose a trajectory-grounded LLM-as-a-Judge pipeline for scalable and reproducible failure attribution, and validate it with human annotation on trajectories, achieving strong agreement (inter-annotator κ=0.61; human-judge κ=0.84). Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures and offer practical guidance for building more reliable long-horizon agents. We release our project website at \href{https://xwang2775.github.io/horizon-leaderboard/}{HORIZON Leaderboard} and welcome contributions from the community.

Related papers

• DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents [29.982261116765162]
  DynSTEER is a dynamic stage-wise framework for agent trajectory evaluation.<n>It segments rollouts at key execution nodes and adapts its multi-tier review strategy based on stage-level results.<n>It compiles a path-tolerant milestone graph from public task views to preserve diverse legal paths without reference leakage.
  arXiv  Detail & Related papers  (2026-09-13T16:18:03Z)
• HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning [51.95194664412812]
  Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks.<n>We propose Hierarchical Planning and Information Folding (HIPIF) for long-horizon LLM agent learning.<n>HIPIF trains the agent end-to-end to organize long-horizon execution around explicit subgoals while folding completed subgoal histories to reduce long-context interference.
  arXiv  Detail & Related papers  (2026-06-09T07:35:14Z)
• When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution [17.020655152761346]
  LongAct is a benchmark designed to evaluate planning-level autonomy in long-horizon household tasks.<n>HoloMind is a VLM-driven agent with a DAG-based long-horizon hierarchical planner.
  arXiv  Detail & Related papers  (2026-05-14T07:47:53Z)
• On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length [62.99653688239127]
  We study the role of task horizon length in shaping training dynamics.<n>We find that increasing horizon length alone constitutes a training bottleneck.<n>We demonstrate that horizon reduction is a key principle to address this limitation.
  arXiv  Detail & Related papers  (2026-05-04T13:25:05Z)
• AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems [0.0]
  We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent AI systems.<n>AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals.<n>Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.
  arXiv  Detail & Related papers  (2026-03-16T00:46:44Z)
• MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning [42.779100789823055]
  We propose MAGE, a Multi-scale Autoregressive GEneration-based offline RL method.<n>MAGE incorporates a condition-guided multi-scale autoencoder to learn hierarchical trajectory representations.<n>Experiments show that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance.
  arXiv  Detail & Related papers  (2026-02-27T07:56:33Z)
• From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems [13.588125015284257]
  We propose CHIEF, a novel framework that transforms chaotic trajectories into a structured hierarchical causal graph.<n> Experiments on Who&When benchmark show that CHIEF outperforms eight strong and state-of-the-art baselines on both agent- and step-level accuracy.
  arXiv  Detail & Related papers  (2026-02-27T06:08:42Z)
• Alignment in Time: Peak-Aware Orchestration for Long-Horizon Agentic Systems [2.5424331328233207]
  We introduce APEMO (Affect-aware Peak-End Modulation for Orchestration), a runtime scheduling layer.<n>APEMO detects trajectory instability through behavioral proxies and targets repairs at critical segments, such as peak moments and endings.<n>Our results reframe alignment as a temporal control problem, offering a resilient engineering pathway for the development of long-horizon agentic systems.
  arXiv  Detail & Related papers  (2026-02-20T00:16:07Z)
• LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces [65.11019654023978]
  LongCLI-Bench is a benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks.<n>We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world tasks.<n>Experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench.
  arXiv  Detail & Related papers  (2026-02-15T23:12:57Z)
• Agentic Spatio-Temporal Grounding via Collaborative Reasoning [80.83158605034465]
  Temporal Video Grounding aims to retrieve thetemporal tube of a target object or person in a video given a text query.<n>We propose the Agentic Spatio-Temporal Grounder (ASTG) framework for the task of STVG towards an open-world and training-free scenario.<n>Specifically, two specialized agents SRA (Spatial Reasoning Agent) and TRA (Temporal Reasoning Agent) constructed leveraging on modern Multimoal Large Language Models (MLLMs)<n>Experiments on popular benchmarks demonstrate the superiority of the proposed approach where it outperforms existing weakly-supervised and zero-shot approaches by a margin
  arXiv  Detail & Related papers  (2026-02-10T10:16:27Z)
• Toward Ultra-Long-Horizon Agentic Science: Cognitive Accumulation for Machine Learning Engineering [59.18634614089481]
  We present ML-Master 2.0, an autonomous agent that masters ultra-long-horizon machine learning engineering (MLE)<n>By reframing context management as a process of cognitive accumulation, our approach introduces Hierarchical Cognitive Caching (HCC)<n>HCC allows agents to decouple immediate execution from long-term experimental strategy.<n>In evaluations on OpenAI's MLE-Bench under 24-hour budgets, ML-Master 2.0 achieves a state-of-the-art medal rate of 56.44%.
  arXiv  Detail & Related papers  (2026-01-15T13:52:04Z)
• Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale [53.08403177911567]
  textitCraftax-MA is an extension of the popular open-ended RL environment, Craftax.<n>textitCraftax-Coop introduces heterogeneous agents, trading and more mechanics that require complex cooperation among agents for success.
  arXiv  Detail & Related papers  (2025-11-07T01:09:36Z)
• UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios [63.67884284105684]
  We introduce textbfUltraHorizon, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges.<n>Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules.<n>Our experiments reveal that LLM-agents consistently underperform in these settings, whereas human participants achieve higher scores.
  arXiv  Detail & Related papers  (2025-09-26T02:04:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.