EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2606.17680v1
• Date: Tue, 16 Jun 2026 08:48:09 GMT
• Title: EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning
• Abstract summary: Reinforcement learning (RL) has emerged as a powerful paradigm for training Large Language Models (LLMs) as agents.<n>We propose EnvRL, a framework that incorporates environment dynamics learning into agentic RL via two auxiliary objectives.
• Score: 78.62829041672663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has emerged as a powerful paradigm for training Large Language Models (LLMs) as agents. However, conventional RL methods for long-horizon agentic tasks often struggle with sparse outcome rewards. Intuitively, this overlooks the rich environment dynamics information contained in rollout interaction trajectories. We argue that the interaction experience inherently serves as an implicit supervision signal, reveals the underlying transition mechanisms of the environment, and enables the agent to construct a more accurate internal model of the environment.. Therefore, in this work, we investigate how to leverage this additional signal to improve policy learning. Specifically, we propose EnvRL, a framework that incorporates environment dynamics learning into agentic RL via two auxiliary objectives: state prediction and inverse dynamics. By jointly optimizing with the primary RL objective, we encourage the agent to internalize environment dynamics from its own interaction experience. Extensive experiments on two long-horizon agentic benchmarks demonstrate that EnvRL achieves significant improvements on success-rates over RL-only baselines, e.g., when trained with GRPO, lifting Qwen-2.5-1.5B-Instruct from 72.8% to 77.4% on ALFWorld, and from 56.8% to 67.0% on WebShop.

Related papers

• Learning Generalizable Behaviors for Terminal Agents [72.62493560701414]
  We propose a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization.<n>Using this recipe, our RL-trained agent account the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks.
  arXiv  Detail & Related papers  (2026-08-23T22:34:40Z)
• RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System [52.3348044324205]
  We propose RLAnything, a reinforcement learning framework that forges environment, policy, and reward models through closed-loop optimization.<n>Specifically, the policy is trained with integrated feedback from step-wise and outcome signals.<n>Our theory-motivated automatic environment adaptation improves training for both the reward and policy models.
  arXiv  Detail & Related papers  (2026-02-02T18:59:04Z)
• Internalizing World Models via Self-Play Finetuning for Agentic RL [65.96875390986655]
  Large Language Models (LLMs) as agents often struggle in out-of-distribution (OOD) scenarios.<n>We show how to encode this world model by decomposing it into two components: state representation and transition modeling.<n>We introduce SPA, a simple reinforcement learning framework that cold-starts the policy via a Self-Play supervised finetuning stage to learn the world model.
  arXiv  Detail & Related papers  (2025-10-16T18:03:39Z)
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning [129.44038804430542]
  We introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL.<n>We propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization.<n>Our agents match or surpass commercial models on 27 tasks across diverse environments.
  arXiv  Detail & Related papers  (2025-09-10T16:46:11Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.96848846966087]
  Training large language models (LLMs) as interactive agents presents unique challenges.<n>While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored.<n>We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL [26.169030913260084]
  We present Ego-Foresight, a self-supervised method for disentangling agent and environment based on motion and prediction.<n>Our main finding is self-supervised agent-awareness by visuomotor prediction of the agent improves sample-efficiency and performance of the underlying RL algorithm.
  arXiv  Detail & Related papers  (2024-05-27T13:32:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.