Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2607.07178v1
• Date: Wed, 08 Jul 2026 09:13:05 GMT
• Title: Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning
• Abstract summary: We show that different tasks can exhibit exploration-exploitation pace mismatch.<n>This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes.<n>Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic Large Language Model (LLM) tasks.
• Score: 82.58758211120319
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent breakthroughs of Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts largely focus on single-task settings, whereas real-world deployment necessitates a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: different tasks can exhibit exploration-exploitation pace mismatch. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can, in turn, push easier tasks back toward high-entropy exploration. This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on the multi-task agentic benchmarks demonstrate that the proposed EPPO yields results superior to its counterparts.

Related papers

• Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling [11.13427204975652]
  Multi-task reinforcement learning aims to train a single agent to efficiently optimize performance across multiple tasks simultaneously.<n> jointly optimizing all tasks often yields imbalanced learning: agents quickly solve easy tasks but learn slowly on harder ones.<n>Standard MTRL allocates an equal number of environment interactions to each task, which over-allocates data to easy tasks.<n>DRATS is an algorithm that adaptively prioritizes sampling tasks furthest from being solved.
  arXiv  Detail & Related papers  (2026-05-14T04:22:24Z)
• Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning [15.393743659727926]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in knowledge acquisition, reasoning, and tool use.<n>This paper introduces a novel approach that transforms multi-turn task planning into single-turn task reasoning problems.
  arXiv  Detail & Related papers  (2025-09-24T23:47:36Z)
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners [60.75160178669076]
  We show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online reinforcement learning.<n>We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL.
  arXiv  Detail & Related papers  (2025-05-29T06:41:45Z)
• Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning [70.96345405979179]
  The purpose of offline multi-task reinforcement learning (MTRL) is to develop a unified policy applicable to diverse tasks without the need for online environmental interaction. variations in task content and complexity pose significant challenges in policy formulation. We introduce the Harmony Multi-Task Decision Transformer (HarmoDT), a novel solution designed to identify an optimal harmony subspace of parameters for each task.
  arXiv  Detail & Related papers  (2024-11-02T05:49:14Z)
• Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning [61.294110816231886]
  We introduce a sparse, reusable, and flexible policy, Sparse Diffusion Policy (SDP) SDP selectively activates experts and skills, enabling efficient and task-specific learning without retraining the entire model. Demos and codes can be found in https://forrest-110.io/sparse_diffusion_policy/.
  arXiv  Detail & Related papers  (2024-07-01T17:59:56Z)
• Provable Benefits of Multi-task RL under Non-Markovian Decision Making Processes [56.714690083118406]
  In multi-task reinforcement learning (RL) under Markov decision processes (MDPs), the presence of shared latent structures has been shown to yield significant benefits to the sample efficiency compared to single-task RL. We investigate whether such a benefit can extend to more general sequential decision making problems, such as partially observable MDPs (POMDPs) and more general predictive state representations (PSRs) We propose a provably efficient algorithm UMT-PSR for finding near-optimal policies for all PSRs, and demonstrate that the advantage of multi-task learning manifests if the joint model class of PSR
  arXiv  Detail & Related papers  (2023-10-20T14:50:28Z)
• Continual Task Allocation in Meta-Policy Network via Sparse Prompting [42.386912478509814]
  We show how to train a generalizable meta-policy by continually learning a sequence of tasks. We address it by "Continual Task Allocation via Sparse Prompting (CoTASP)" In experiments, CoTASP achieves a promising plasticity-stability trade-off without storing or replaying any past tasks' experiences.
  arXiv  Detail & Related papers  (2023-05-29T03:36:32Z)
• Learning Complex Teamwork Tasks Using a Given Sub-task Decomposition [11.998708550268978]
  We propose an approach which uses an expert-provided decomposition of a task into simpler multi-agent sub-tasks. In each sub-task, a subset of the entire team is trained to acquire sub-task-specific policies. The sub-teams are then merged and transferred to the target task, where their policies are collectively fine-tuned to solve the more complex target task.
  arXiv  Detail & Related papers  (2023-02-09T21:24:56Z)
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies [57.27944046925876]
  We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
  arXiv  Detail & Related papers  (2020-01-01T17:34:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.