Maximum-Entropy Exploration with Future State-Action Visitation Measures

• URL: http://arxiv.org/abs/2603.18965v1
• Date: Thu, 19 Mar 2026 14:30:08 GMT
• Title: Maximum-Entropy Exploration with Future State-Action Visitation Measures
• Abstract summary: We study intrinsic rewards proportional to the entropy of the discounted distribution of state-action features visited during future time steps.<n>We show that the expected sum of these intrinsic rewards is a lower bound on the entropy of the discounted distribution of state-action features visited in trajectories.<n>Experiments highlight that the new objective leads to improved visitation of features within individual trajectories, in exchange for slightly reduced visitation of features in expectation over different trajectories.
• Score: 1.3159512679346688
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Maximum entropy reinforcement learning motivates agents to explore states and actions to maximize the entropy of some distribution, typically by providing additional intrinsic rewards proportional to that entropy function. In this paper, we study intrinsic rewards proportional to the entropy of the discounted distribution of state-action features visited during future time steps. This approach is motivated by two results. First, we show that the expected sum of these intrinsic rewards is a lower bound on the entropy of the discounted distribution of state-action features visited in trajectories starting from the initial states, which we relate to an alternative maximum entropy objective. Second, we show that the distribution used in the intrinsic reward definition is the fixed point of a contraction operator and can therefore be estimated off-policy. Experiments highlight that the new objective leads to improved visitation of features within individual trajectories, in exchange for slightly reduced visitation of features in expectation over different trajectories, as suggested by the lower bound. It also leads to improved convergence speed for learning exploration-only agents. Control performance remains similar across most methods on the considered benchmarks.

Related papers

• Rethinking Exploration in RLVR: From Entropy Regularization to Refinement via Bidirectional Entropy Modulation [33.15281300187744]
  Reinforcement learning with rewards has advanced the reasoning capabilities of large language models.<n>It faces a limitation termed textitrestricted exploration, where the policy rapidly converges to a narrow set of solutions.<n>We propose textbfAsymGRPO, an exploratory framework that explicitly decouples the modulation of positive and negative rollouts.
  arXiv  Detail & Related papers  (2026-04-06T17:42:38Z)
• Maximum Entropy Exploration Without the Rollouts [5.008597638379228]
  principled iteration of the exploration problem is to find policies that maximize the entropy of their induced steady-state visitation distribution.<n>In this work, we consider an intrinsic average-reward in which the reward is derived from the visitation distribution itself, so that the optimal policy maximizes steady-state entropy.<n>This insight leads to a novel algorithm for solving the maximum entropy exploration problem, EVE, which avoids explicit rollouts and distribution estimation.
  arXiv  Detail & Related papers  (2026-03-12T18:00:03Z)
• Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinement [51.54933696252104]
  We introduce Iterative Reward-Guided Refinement (IterRef), a novel test-time scaling method tailored to discrete diffusion.<n>We formalize this process within a Multiple-Try Metropolis framework, proving convergence to the reward-aligned distribution.<n>IterRef achieves striking gains under low compute budgets, far surpassing prior state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-04T02:33:23Z)
• Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning [56.87989363424]
  We show that a low-rank structure naturally emerges in the shifted successor measure.<n>We quantify the amount of shift needed for effective low-rank approximation and estimation.
  arXiv  Detail & Related papers  (2025-09-05T15:48:20Z)
• Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures [1.3159512679346688]
  We propose a novel approach in which the intrinsic reward function is the relative entropy of the discounted distribution of states and actions.<n>Existing algorithms can therefore be adapted to learn this fixed point off-policy and to compute the intrinsic rewards.<n>We show that resulting policies have good state-action space coverage and achieve high-performance control.
  arXiv  Detail & Related papers  (2024-12-09T16:56:06Z)
• The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enough [40.82741665804367]
  We study a simple approach of maximizing the entropy over observations in place true latent states. We show how knowledge of the latter can be exploited to compute a regularization of the observation entropy to improve principled performance.
  arXiv  Detail & Related papers  (2024-06-18T17:00:13Z)
• Reward Collapse in Aligning Large Language Models [73.78408135192471]
  We study the phenomenon of textitreward collapse', an empirical observation where the prevailing ranking-based approach results in an textitidentical reward distribution.<n>Our experimental results suggest that our proposed prompt-aware utility functions significantly alleviate reward collapse during the training of reward models.
  arXiv  Detail & Related papers  (2023-05-28T02:12:00Z)
• Complex behavior from intrinsic motivation to occupy action-state path space [0.0]
  We propose that the goal of behavior is maximizing future occupancy of paths of actions and states. According to this occupancy principle, rewards are the means to occupy path space, not the goal per se. We show that complex behaviors such as dancing', hide-and-seek and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space.
  arXiv  Detail & Related papers  (2022-05-20T17:32:41Z)
• Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes [70.80716221080118]
  The paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning. The OS-GPTD approach is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs. To alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme.
  arXiv  Detail & Related papers  (2021-12-01T23:15:09Z)
• Action Redundancy in Reinforcement Learning [54.291331971813364]
  We show that transition entropy can be described by two terms; namely, model-dependent transition entropy and action redundancy. Our results suggest that action redundancy is a fundamental problem in reinforcement learning.
  arXiv  Detail & Related papers  (2021-02-22T19:47:26Z)
• A maximum-entropy approach to off-policy evaluation in average-reward MDPs [54.967872716145656]
  This work focuses on off-policy evaluation (OPE) with function approximation in infinite-horizon undiscounted Markov decision processes (MDPs) We provide the first finite-sample OPE error bound, extending existing results beyond the episodic and discounted cases. We show that this results in an exponential-family distribution whose sufficient statistics are the features, paralleling maximum-entropy approaches in supervised learning.
  arXiv  Detail & Related papers  (2020-06-17T18:13:37Z)
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward [64.24528565312463]
  This paper tackles the challenge of using belief-based rewards for a deep RL agent. We derive the exact error between negative entropy and the expected prediction reward. This insight provides theoretical motivation for several fields using prediction rewards.
  arXiv  Detail & Related papers  (2020-05-11T08:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.