Hell or High Water: Evaluating Agentic Recovery from External Failures

• URL: http://arxiv.org/abs/2508.11027v1
• Date: Thu, 14 Aug 2025 19:21:09 GMT
• Title: Hell or High Water: Evaluating Agentic Recovery from External Failures
• Abstract summary: We develop a specialized agentic planning benchmark to study how well language agents search for alternative ways to achieve their goals.<n>Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable.<n>Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback.
• Score: 19.402114652658206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As language model agents are applied to real world problems of increasing complexity, they will be expected to formulate plans across large search spaces. If those plans fail for reasons beyond their control, how well do language agents search for alternative ways to achieve their goals? We devise a specialized agentic planning benchmark to study this question. Each planning problem is solved via combinations of function calls. The agent searches for relevant functions from a set of over four thousand possibilities, and observes environmental feedback in the form of function outputs or error messages. Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable. At the same time, even with the introduction of these failures, we guarantee that the task remains solvable. Ideally, an agent's performance on the planning task should not be affected by the presence of external failures. Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback. While state-of-the-art models are often able to identify the correct function to use in the right context, they struggle to adapt to feedback from the environment and often fail to pursue alternate courses of action, even when the search space is artificially restricted. We provide a systematic analysis of the failures of both open-source and commercial models, examining the effects of search space size, as well as the benefits of scaling model size in our setting. Our analysis identifies key challenges for current generative models as well as promising directions for future work.

Related papers

• Recovering Wasted Compute in Autoresearch Agents [35.718169903054736]
  We study the modeling pipeline at the core of autoresearch systems and identify common failure modes.<n>We find that large gains in autoresearch agent performance are achievable through agentic design alone.
  arXiv  Detail & Related papers  (2026-08-11T03:15:08Z)
• TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints [0.0]
  We introduce TRIAGE, a framework for testing language models for metacognitive control.<n>We find that current language models exhibit substantial gaps in prospective metacognitive control.
  arXiv  Detail & Related papers  (2026-05-13T12:10:05Z)
• FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments [60.3427704389541]
  Large Language Models are being increasingly deployed as the decision-making core of autonomous agents.<n>Yet, in conversational benchmarks, these agents frequently fail due to the cascading effects of incorrect decision-making.<n>We present the Failure-Aware Meta-Agentic framework to tackle these challenges.
  arXiv  Detail & Related papers  (2026-04-28T02:21:53Z)
• Exploration and Exploitation Errors Are Measurable for Language Model Agents [52.95061000593404]
  Language Model (LM) agents are increasingly used in complex openended decision tasks.<n>A core requirement is the ability to both explore the problem space and exploit acquired knowledge effectively.<n>We design controllable environments inspired by practical embodied AI scenarios.
  arXiv  Detail & Related papers  (2026-04-14T17:59:57Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding [50.35012849818872]
  Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks.<n>We propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding.<n>Our experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment.
  arXiv  Detail & Related papers  (2026-01-14T14:09:11Z)
• Current Agents Fail to Leverage World Model as Tool for Foresight [61.82522354207919]
  Generative world models offer a promising remedy: agents could use them to foresee outcomes before acting.<n>This paper empirically examines whether current agents can leverage such world models as tools to enhance their cognition.
  arXiv  Detail & Related papers  (2026-01-07T13:15:23Z)
• Limits of Emergent Reasoning of Large Language Models in Agentic Frameworks for Deterministic Games [2.357397994148727]
  Large Reasoning Models (LRMs) undergo a collapse in performance on solving puzzles beyond certain perplexity thresholds.<n>We provide a large language model (LLM) with an environment interface for Tower of Hanoi problems.<n>We observe that access to an environment interface does not delay or eradicate performance collapse.
  arXiv  Detail & Related papers  (2025-10-12T23:48:16Z)
• UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios [63.67884284105684]
  We introduce textbfUltraHorizon, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges.<n>Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules.<n>Our experiments reveal that LLM-agents consistently underperform in these settings, whereas human participants achieve higher scores.
  arXiv  Detail & Related papers  (2025-09-26T02:04:00Z)
• Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories [18.129031749321058]
  Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks.<n>Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored.<n>We present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents.
  arXiv  Detail & Related papers  (2025-06-23T16:34:52Z)
• ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks [54.52092001110694]
  ThinkGeo is a benchmark designed to evaluate tool-augmented agents on remote sensing tasks via structured tool use and multi-step planning.<n>Inspired by tool-interaction paradigms, ThinkGeo includes human-curated queries spanning a wide range of real-world applications.<n>Our analysis reveals notable disparities in tool accuracy and planning consistency across models.
  arXiv  Detail & Related papers  (2025-05-29T17:59:38Z)
• Self-Corrective Task Planning by Inverse Prompting with Large Language Models [9.283971287618261]
  We introduce InversePrompt, a novel self-corrective task planning approach.<n>Our method incorporates reasoning steps to provide clear, interpretable feedback.<n>Results on benchmark datasets show an average 16.3% higher success rate over existing LLM-based task planning methods.
  arXiv  Detail & Related papers  (2025-03-10T13:35:51Z)
• Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring [5.482898079941062]
  We present AutoRefine, a method that leverages reinforcement learning for targeted fine-tuning.<n>We demonstrate the method for a problem arising in algorithmic hiring platforms where linguistic biases influence a recommendation system.<n>Our model detects and regulates biases in job descriptions to meet diversity and fairness criteria.
  arXiv  Detail & Related papers  (2025-01-13T13:36:17Z)
• KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents [52.34892973785117]
  Large Language Models (LLMs) have demonstrated great potential in complex reasoning tasks, yet they fall short when tackling more sophisticated challenges.<n>This inadequacy primarily stems from the lack of built-in action knowledge in language agents.<n>We introduce KnowAgent, a novel approach designed to enhance the planning capabilities of LLMs by incorporating explicit action knowledge.
  arXiv  Detail & Related papers  (2024-03-05T16:39:12Z)
• Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models [58.57279229066477]
  We study how language models (LMs) solve retrieval tasks in diverse situations. We introduce ORION, a collection of structured retrieval tasks spanning six domains. We find that LMs internally decompose retrieval tasks in a modular way.
  arXiv  Detail & Related papers  (2023-12-13T18:36:43Z)
• AI planning in the imagination: High-level planning on learned abstract search spaces [68.75684174531962]
  We propose a new method, called PiZero, that gives an agent the ability to plan in an abstract search space that the agent learns during training. We evaluate our method on multiple domains, including the traveling salesman problem, Sokoban, 2048, the facility location problem, and Pacman.
  arXiv  Detail & Related papers  (2023-08-16T22:47:16Z)
• Asking Before Acting: Gather Information in Embodied Decision Making with Language Models [20.282749796376063]
  We show that Large Language Models (LLMs) encounter challenges in efficiently gathering essential information in unfamiliar environments. We propose textitAsking Before Acting (ABA), a method that empowers the agent to proactively inquire with external sources for pertinent information using natural language. We conduct extensive experiments involving a spectrum of environments including text-based household everyday tasks, robot arm manipulation tasks, and real world open domain image based embodied tasks.
  arXiv  Detail & Related papers  (2023-05-25T04:05:08Z)
• Reflexion: Language Agents with Verbal Reinforcement Learning [44.85337947858337]
  Reflexion is a novel framework to reinforce language agents not by updating weights, but through linguistic feedback. It is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals. For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%.
  arXiv  Detail & Related papers  (2023-03-20T18:08:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.