Exploration and Exploitation Errors Are Measurable for Language Model Agents

• URL: http://arxiv.org/abs/2604.13151v1
• Date: Tue, 14 Apr 2026 17:59:57 GMT
• Title: Exploration and Exploitation Errors Are Measurable for Language Model Agents
• Abstract summary: Language Model (LM) agents are increasingly used in complex openended decision tasks.<n>A core requirement is the ability to both explore the problem space and exploit acquired knowledge effectively.<n>We design controllable environments inspired by practical embodied AI scenarios.
• Score: 52.95061000593404
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Language Model (LM) agents are increasingly used in complex open-ended decision-making tasks, from AI coding to physical AI. A core requirement in these settings is the ability to both explore the problem space and exploit acquired knowledge effectively. However, systematically distinguishing and quantifying exploration and exploitation from observed actions without access to the agent's internal policy remains challenging. To address this, we design controllable environments inspired by practical embodied AI scenarios. Each environment consists of a partially observable 2D grid map and an unknown task Directed Acyclic Graph (DAG). The map generation can be programmatically adjusted to emphasize exploration or exploitation difficulty. To enable policy-agnostic evaluation, we design a metric to quantify exploration and exploitation errors from agent's actions. We evaluate a variety of frontier LM agents and find that even state-of-the-art models struggle on our task, with different models exhibiting distinct failure modes. We further observe that reasoning models solve the task more effectively and show both exploration and exploitation can be significantly improved through minimal harness engineering. We release our code \href{https://github.com/jjj-madison/measurable-explore-exploit}{here}.

Related papers

• Multi-Agent LLMs Fail to Explore Each Other [34.74439361133657]
  We show that modern large language model (LLM) agents fail to explore effectively when interacting with one another.<n>We introduce Multi- Agent Contextual Exploration (MACE), a lightweight framework that explicitly promotes exploration through structured peer selection.
  arXiv  Detail & Related papers  (2026-07-13T08:34:05Z)
• Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization [14.566152113959555]
  A key limitation of existing methods is that they typically employ undifferentiated exploration strategies.<n>We propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high.
  arXiv  Detail & Related papers  (2026-05-09T14:44:18Z)
• Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration [12.531650952835493]
  In this paper, we propose a new paradigm that explicitly separates exploration from exploitation and bypasses RL during the exploration phase.<n>By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard Atari benchmarks.<n>We demonstrate that the discovered trajectories can be distilled into deployable policies using existing supervised backward learning algorithms.
  arXiv  Detail & Related papers  (2026-03-23T17:56:52Z)
• What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding [50.35012849818872]
  Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks.<n>We propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding.<n>Our experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment.
  arXiv  Detail & Related papers  (2026-01-14T14:09:11Z)
• Hell or High Water: Evaluating Agentic Recovery from External Failures [19.402114652658206]
  We develop a specialized agentic planning benchmark to study how well language agents search for alternative ways to achieve their goals.<n>Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable.<n>Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback.
  arXiv  Detail & Related papers  (2025-08-14T19:21:09Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks.<n>However, they still struggle with problems requiring multi-step decision-making and environmental feedback.<n>We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• Latent Exploration for Reinforcement Learning [87.42776741119653]
  In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. We propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network.
  arXiv  Detail & Related papers  (2023-05-31T17:40:43Z)
• Agent Spaces [0.0]
  We define exploration as the act of modifying an agent to itself be explorative. We show that many important structures in Reinforcement Learning are well behaved under the topology induced by convergence in the agent space.
  arXiv  Detail & Related papers  (2021-11-11T01:12:17Z)
• Deep Reinforcement Learning for Adaptive Exploration of Unknown Environments [6.90777229452271]
  We develop an adaptive exploration approach to trade off between exploration and exploitation in one single step for UAVs. The proposed approach uses a map segmentation technique to decompose the environment map into smaller, tractable maps. The results demonstrate that our proposed approach is capable of navigating through randomly generated environments and covering more AoI in less time steps compared to the baselines.
  arXiv  Detail & Related papers  (2021-05-04T16:29:44Z)
• Batch Exploration with Examples for Scalable Robotic Reinforcement Learning [63.552788688544254]
  Batch Exploration with Examples (BEE) explores relevant regions of the state-space guided by a modest number of human provided images of important states. BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot.
  arXiv  Detail & Related papers  (2020-10-22T17:49:25Z)
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships [52.72020203771489]
  We investigate target-driven visual navigation using deep reinforcement learning (DRL) in 3D indoor scenes. Our proposed method combines visual features and 3D spatial representations to learn navigation policy. Our experiments, performed in the AI2-THOR, show that our model outperforms the baselines in both SR and SPL metrics.
  arXiv  Detail & Related papers  (2020-04-29T08:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.