Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

• URL: http://arxiv.org/abs/2606.21271v1
• Date: Fri, 19 Jun 2026 09:47:38 GMT
• Title: Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization
• Abstract summary: Sparse rewards pose a central challenge in reinforcement learning, since agents receive no informative signal until they reach their goal.<n>We present a method to pre-train transferable exploration policies that rapidly adapt to sparse rewards at downstream task time.
• Score: 31.035684025590417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse rewards pose a central challenge in reinforcement learning, since agents receive no informative signal until they reach their goal. Intrinsic-reward methods address this issue by optimizing non-stationary objectives such as novelty, prediction error, or skill diversity, thereby injecting a supervision signal into the problem. While effective, these methods often require that the extrinsic (sparse) reward can be evaluated -- either online or during offline relabeling of the stored transitions. This limitation is particularly vexing for multi-task, meta-, and continual reinforcement learning, where agents' interactions with the environment are usually reward-free. In this work, we present a method to pre-train transferable exploration policies that rapidly adapt to sparse rewards at downstream task time. Our objective maximizes state-space covering for the occupancy measure, and can be framed in terms of entropy maximization. Its algorithmic implementation, ROVER, leverages recent advances on the operatorial formulation of RL to estimate occupancy with a learned resolvent world model, bypassing common hurdles associated with density and entropy estimation. ROVER further introduces a virtual "sink" state for unexplored regions, balancing coverage of known states with expansion into unseen ones and preventing cyclic expansion-collapse behavior during learning. In tabular and pixel-based sparse navigation tasks, ROVER produces more uniform aggregate coverage and stronger initializations for downstream tasks than standard reward-free baselines.

Related papers

• Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation [91.20850436220267]
  We introduce the Referring-Aware Visuomotor Policy (ReV)<n>ReV incorporates sparse referring points provided by a human or a high-level reasoning planner.<n>It is trained only by applying targeted perturbations to expert demonstrations.
  arXiv  Detail & Related papers  (2026-04-07T07:41:11Z)
• Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies [31.52910494173408]
  Diffusion-based robot navigation policies can generate multi-modal trajectories directly from the robot's visual observations.<n>We propose a reinforcement learning framework tailored for diffusion-based navigation.<n>Our approach improves the Success Rate from 52.0% to 58.7% and SPL from 0.49 to 0.54 on unseen scenes, while reducing collision frequency.
  arXiv  Detail & Related papers  (2026-03-13T10:14:32Z)
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning [49.60927398960061]
  We propose a neuro-symbolic Deep RL approach that integrates background symbolic knowledge to improve sample efficiency.<n>Online reasoning is performed to guide the training process through two mechanisms.<n>We show improved performance over a state-of-the-art reward machine baseline.
  arXiv  Detail & Related papers  (2026-01-06T09:28:53Z)
• Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning [51.07663354001582]
  Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task.<n>We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches.<n>We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods.
  arXiv  Detail & Related papers  (2025-12-01T15:56:00Z)
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning [63.73629127832652]
  We introduce TD-JEPA, which leverages TD-based latent-predictive representations into unsupervised RL.<n> TD-JEPA trains explicit state and task encoders, a policy-conditioned multi-step predictor, and a set of parameterized policies directly in latent space.<n> Empirically, TD-JEPA matches or outperforms state-of-the-art baselines on locomotion, navigation, and manipulation tasks across 13 datasets.
  arXiv  Detail & Related papers  (2025-10-01T10:21:18Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning [0.0]
  We introduce DQInit, a method that adapts value function initialization to deep reinforcement learning.<n>DQInit reuses compact Q-values extracted from previously solved tasks as a transferable knowledge base.<n>It employs a knownness-based mechanism to softly integrate these transferred values into underexplored regions and gradually shift toward the agent's learned estimates.
  arXiv  Detail & Related papers  (2025-08-12T18:32:08Z)
• Imagine Beyond! Distributionally Robust Auto-Encoding for State Space Coverage in Online Reinforcement Learning [15.884955846999246]
  Goal-Conditioned Reinforcement Learning (GCRL) enables agents to autonomously acquire diverse behaviors.<n>In the online setting, where agents learn representations while exploring, the latent space evolves with the agent's policy.
  arXiv  Detail & Related papers  (2025-05-23T12:43:55Z)
• Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models [71.34520793462069]
  Unsupervised reinforcement learning (RL) aims at pre-training agents that can solve a wide range of downstream tasks in complex environments.<n>We introduce a novel algorithm regularizing unsupervised RL towards imitating trajectories from unlabeled behavior datasets.<n>We demonstrate the effectiveness of this new approach in a challenging humanoid control problem.
  arXiv  Detail & Related papers  (2025-04-15T10:41:11Z)
• Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces [13.544995860887298]
  In many reinforcement learning (RL) applications, we want policies that reach desired states and keep the controlled system within an acceptable region. In this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies. We introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space.
  arXiv  Detail & Related papers  (2023-06-02T20:01:09Z)
• Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows [58.762959061522736]
  offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. We build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms.
  arXiv  Detail & Related papers  (2022-11-20T21:57:10Z)
• Understanding and Preventing Capacity Loss in Reinforcement Learning [28.52122927103544]
  We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents. Capacity loss occurs in a range of RL agents and environments, and is particularly damaging to performance in sparse-reward tasks.
  arXiv  Detail & Related papers  (2022-04-20T15:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.