StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction

• URL: http://arxiv.org/abs/2605.06642v1
• Date: Thu, 07 May 2026 17:51:16 GMT
• Title: StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction
• Abstract summary: StraTA is a framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning.<n>Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance.
• Score: 76.50439715778869
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.

Related papers

• Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning [76.93011742289768]
  Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories.<n>This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills.<n>We propose Strategy-Guided Policy Optimization, which replaces instance-level trajectory imitation with reusable strategy distillation.
  arXiv  Detail & Related papers  (2026-06-23T02:14:12Z)
• Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR [53.27792011950384]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models.<n>We propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR.<n>Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts.
  arXiv  Detail & Related papers  (2026-05-15T08:22:59Z)
• GAGPO: Generalized Advantage Grouped Policy Optimization [13.085463559931156]
  We propose a critic-free reinforcement learning method for precise, step-aligned temporal credit assignment.<n>Experiments on ALFWorld and WebShop show that GAGPO outperforms strong reinforcement learning baselines.
  arXiv  Detail & Related papers  (2026-05-13T09:10:03Z)
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration [51.98616048282804]
  Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding.<n>We propose Strategy-Guided Exploration (SGE) to shift exploration from low-level actions to higher-level language strategies.
  arXiv  Detail & Related papers  (2026-03-02T16:28:39Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Reflecting with Two Voices: A Co-Adaptive Dual-Strategy Framework for LLM-Based Agent Decision Making [24.534365665776672]
  Large language model (LLM) agents often rely on external demonstrations or retrieval-augmented planning.<n>We propose DuSAR - a demonstration-free framework that enables a single frozen LLM to perform co-adaptive reasoning.<n>On ALFWorld and Mind2Web, DuSAR achieves state-of-the-art performance with open-source LLMs.
  arXiv  Detail & Related papers  (2025-12-09T08:44:59Z)
• SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph [15.303565124424635]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, enabling language agents to excel at single-turn tasks.<n>Their application to complex, multi-step, and long-horizon tasks remains challenging.<n>We propose SALT, a framework that provides a finer-grained advantage assignment, derived solely from outcome rewards.
  arXiv  Detail & Related papers  (2025-10-22T20:50:24Z)
• Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming [45.95972813586392]
  Existing red-teaming approaches mainly rely on manually crafted attack strategies or static models trained offline.<n>We propose Genesis, a novel agentic framework composed of three modules: Attacker, Scorer, and Strategist.<n>Our framework discovers novel strategies and consistently outperforms existing attack baselines.
  arXiv  Detail & Related papers  (2025-10-21T05:49:37Z)
• AutoDAN-Reasoning: Enhancing Strategies Exploration based Jailbreak Attacks with Test-Time Scaling [54.47844626555395]
  AutoDAN-Turbo employs a lifelong learning agent to build a rich library of attack strategies from scratch.<n>While highly effective, its test-time generation process involves sampling a strategy and generating a single corresponding attack prompt.<n>We propose to further improve the attack performance of AutoDAN-Turbo through test-time scaling.
  arXiv  Detail & Related papers  (2025-10-06T21:16:09Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization [12.061547251822326]
  Trajectory-based Group Relative Policy Optimization (TGRPO) is an online RL-based training framework for Visual-Language-Action (VLA) models.<n>We show that TGRPO achieves an average success rate of 80.7%, which is 4.2% higher than that of Supervised Fine-Tuning (SFT) and outperforms other representative RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-06-10T04:27:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.