The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents

• URL: http://arxiv.org/abs/2608.06663v1
• Date: Fri, 07 Aug 2026 00:19:48 GMT
• Title: The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
• Abstract summary: We survey 1,547 arXiv papers collected via systematic seed harvest with a disclosed 26.8% bleed filter.<n>We disambiguate three routinely conflated properties: long-horizon, long-context, and long-term memory.<n>Across all categories, we find the same pattern: outcome-only signals grow uninformative as horizons lengthen.
• Score: 7.159691098846351
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier language models solve reasoning problems in a single forward pass that would have been research contributions years ago, yet fail at multi-hour tasks: losing track of earlier decisions, declaring half-finished work done, or drifting from goals. We call this the horizon gap and survey 1,547 arXiv papers (2024-2026) collected via systematic seed harvest with a disclosed 26.8% bleed filter, extended by targeted supplementation. We disambiguate three routinely conflated properties: long-horizon (task property: required steps), long-context (model property: token capacity), and long-term memory (system property: persistence across steps/sessions). We organize the corpus into six categories tracking a long-horizon task's lifecycle -- planning, memory, execution, training, evaluation, and foundations/safety -- crossed with an axis capturing where horizons are carried (within-context, within-task-beyond-context, or cross-task-persistent). Across all categories, we find the same pattern: outcome-only signals grow uninformative as horizons lengthen, and the field's response -- whether process reward models, credit assignment, or trajectory-level diagnostics -- manufactures denser step-level signals. We treat critical and diagnostic literature as first-class threads throughout, arguing that segregating critique from method would routinely split single papers across chapters. We close by naming open measurement problems: decomposing model versus harness capability, managing correlated bias in process-level signals used for both training and evaluation, and whether long-horizon reliability admits general predictive theory.

Related papers

• Training and Evaluating Diffusion Policies with Long Context Lengths [19.39755600541638]
  We first benchmark policy performance as context length is incrementally increased from short to long.<n>We propose a training algorithm to jointly train policies at multiple context lengths.<n>We re-evaluate some previously proposed solutions to long-context imitation learning.
  arXiv  Detail & Related papers  (2026-06-15T09:19:34Z)
• HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning [51.95194664412812]
  Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks.<n>We propose Hierarchical Planning and Information Folding (HIPIF) for long-horizon LLM agent learning.<n>HIPIF trains the agent end-to-end to organize long-horizon execution around explicit subgoals while folding completed subgoal histories to reduce long-context interference.
  arXiv  Detail & Related papers  (2026-06-09T07:35:14Z)
• MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems [69.06764269022925]
  We study how current memory-augmented agents perform in realistic, interference-heavy, long-horizon settings.<n> MINTEval is a benchmark featuring long, highly interconnected contexts with frequently updated information.<n>Overall, MINTEval has 15.6k question-answering pairs over long-horizon contexts averaging 138.8k tokens and extending to 1.8M tokens per instance.
  arXiv  Detail & Related papers  (2026-05-18T15:43:35Z)
• On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length [62.99653688239127]
  We study the role of task horizon length in shaping training dynamics.<n>We find that increasing horizon length alone constitutes a training bottleneck.<n>We demonstrate that horizon reduction is a key principle to address this limitation.
  arXiv  Detail & Related papers  (2026-05-04T13:25:05Z)
• The Conversations Beneath the Code: Triadic Data for Long-Horizon Software Engineering Agents [1.9097591729497536]
  We argue that the canonical instantiation of triadic data is two complementary products: long-horizon expert trajectories captured under stimulated-recall protocols, and simulated cross-functional companies.<n>We argue that this data is capturable in 12-18 months with methods already mature in adjacent fields.
  arXiv  Detail & Related papers  (2026-05-04T05:37:36Z)
• Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks [67.44219836008348]
  We introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet.<n>We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics.<n>We find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements.
  arXiv  Detail & Related papers  (2026-04-27T20:05:41Z)
• The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break [69.91172974854422]
  HORIZON is a cross-domain diagnostic benchmark for analyzing long-horizon failure behaviors in large language model (LLM) agents.<n>We evaluate state-of-the-art (SOTA) agents from multiple model families to study horizon-dependent degradation patterns.<n>Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures.
  arXiv  Detail & Related papers  (2026-04-13T19:11:42Z)
• UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios [63.67884284105684]
  We introduce textbfUltraHorizon, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges.<n>Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules.<n>Our experiments reveal that LLM-agents consistently underperform in these settings, whereas human participants achieve higher scores.
  arXiv  Detail & Related papers  (2025-09-26T02:04:00Z)
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks [229.73714829399802]
  This survey probes the core challenges that the rise of Large Language Models poses for evaluation.<n>We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety.<n>We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics.
  arXiv  Detail & Related papers  (2025-04-26T07:48:52Z)
• Long-horizon Embodied Planning with Implicit Logical Inference and Hallucination Mitigation [7.668848364013772]
  We present ReLEP, a novel framework for Real-time Long-horizon Embodied Planning.<n>ReLEP can complete a wide range of long-horizon tasks without in-context examples by learning implicit logical inference through fine-tuning.
  arXiv  Detail & Related papers  (2024-09-24T01:47:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.