SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents

• URL: http://arxiv.org/abs/2606.12908v1
• Date: Thu, 11 Jun 2026 05:06:50 GMT
• Title: SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
• Abstract summary: Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use.<n>We propose SENTINEL, a failure-driven reinforcement learning framework that turns rollout failures into targeted training tasks.<n>On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Passk metrics.
• Score: 52.758563996979596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass\^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass\^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.

Related papers

• Iron: Intent-Aligned and Retrospective Dual Learning Framework for Enhancing Generalist Virtual Agents [31.811763840603614]
  We introduce Iron, an intent-aligned framework for training GUI agents.<n>Iron employs a stepwise cycle-consistent reward to achieve fine-grained alignment between low-level actions and high-level intents.<n>Experiments demonstrate that Iron-trained generalist agents consistently improve performance on cross-environment and cross-device tasks.
  arXiv  Detail & Related papers  (2026-08-28T03:15:11Z)
• Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents [80.03824805228719]
  A major challenge in developing computer-use agents is collecting large-scale, high-quality trajectories.<n>We propose a data-centric paradigm that turns failed trajectories into agent improvements.<n>We validate this approach with the state-of-the-art OpenCUA-72B model on the OSWorld benchmark.
  arXiv  Detail & Related papers  (2026-06-30T07:44:37Z)
• Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills [12.442806027914097]
  We introduce Socratic-SWE, a closed-loop self-evolution framework that reuses the agent's historical solving traces as a source of training signal.<n>Rather than treating traces only as evidence for reward computation, Socratic-SWE distills them into structured agent skills that summarize recurring failures and effective repair patterns.
  arXiv  Detail & Related papers  (2026-06-05T16:00:17Z)
• Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals [0.0]
  Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks.<n>We present ULEE, an unsupervised meta-learning method that combines an in-context learner with an adversarial goal-generation strategy.
  arXiv  Detail & Related papers  (2026-01-27T17:10:29Z)
• Structured Uncertainty guided Clarification for LLM Agents [126.26213027785813]
  LLM agents extend large language models with tool-calling capabilities, but ambiguous user instructions often lead to incorrect invocations and task failures.<n>We introduce a principled formulation of structured uncertainty over tool-call parameters, modeling joint tool-argument clarification as a POMDP with Expected Value of Perfect Information (EVPI) objective for optimal question selection and aspect-based cost modeling to prevent redundancy.<n>Our SAGE-Agent leverages this structured uncertainty to achieve superior efficiency: increasing coverage on ambiguous tasks by 7-39% while reducing clarification questions by 1.5-2.7$times$ compared to strong prompting and uncertainty-based baselines.
  arXiv  Detail & Related papers  (2025-11-11T21:50:44Z)
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness [49.72591739116668]
  Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs)<n>However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training.<n>We propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework.
  arXiv  Detail & Related papers  (2025-10-10T13:42:03Z)
• Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning [6.314485350935057]
  Reinforcement Learning with Tool-use Rewards is a novel framework that decouples the training process to enable a focused, single-objective optimization of the planning module.<n>Our experiments demonstrate that RLTR achieves an 8%-12% improvement in planning performance compared to end-to-end baselines.<n>This enhanced planning capability, in turn, translates to a 5%-6% increase in the final response quality of the overall agent system.
  arXiv  Detail & Related papers  (2025-08-27T06:19:50Z)
• Self-Challenging Language Model Agents [98.62637336505242]
  We propose the Self-Challenging framework for training an agent on high-quality tasks that are generated by itself.<n>The framework achieves over a two-fold improvement in Llama-3.1-8B-Instruct, despite using only self-generated training data.
  arXiv  Detail & Related papers  (2025-06-02T14:23:33Z)
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners [60.75160178669076]
  We show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online reinforcement learning.<n>We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL.
  arXiv  Detail & Related papers  (2025-05-29T06:41:45Z)
• P2DT: Mitigating Forgetting in task-incremental Learning with progressive prompt Decision Transformer [49.716834343064015]
  Catastrophic forgetting poses a substantial challenge for managing intelligent agents controlled by a large model.<n>We propose a novel solution - the Progressive Prompt Decision Transformer (P2DT)<n>This method enhances a transformer-based model by dynamically appending decision tokens during new task training, thus fostering task-specific policies.
  arXiv  Detail & Related papers  (2024-01-22T02:58:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.