Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2605.06156v2
• Date: Fri, 08 May 2026 08:54:11 GMT
• Title: Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning
• Abstract summary: We propose textitMaximum Entropy Adjoint Matching (ME-AM), a unified framework that addresses limitations within the continuous flow formulation.<n>ME-AM incorporates two mechanisms: (1) a Mirror Descent entropy objective that adjoins the popularity bias to facilitate the extraction of optimal policies from offline datasets, and (2) a textitMixture Behavior Prior that broadens the geometric support to encompass out-of-distribution high-reward regions.
• Score: 12.008878527093364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Integrating expressive generative policies, such as flow-matching models, into offline reinforcement learning (RL) allows agents to capture complex, multi-modal behaviors. While Q-learning with Adjoint Matching (QAM) stabilizes policy optimization via the continuous adjoint method, it remains inherently bound to the fixed behavior distribution. This dependence induces a \textit{popularity bias} that can suppress high-reward actions in low-density regions, and creates a \textit{support binding} that restricts off-manifold exploration. Existing workarounds, such as appending \textit{residual} Gaussian policies, often re-introduce the expressivity bottlenecks associated with unimodal distributions. In this work, we propose \textit{Maximum Entropy Adjoint Matching} (ME-AM), a unified framework that addresses these limitations within the continuous flow formulation. ME-AM incorporates two mechanisms: (1) a Mirror Descent entropy maximization objective that mitigates the popularity bias to facilitate the extraction of optimal policies from offline datasets, and (2) a \textit{Mixture Behavior Prior} that broadens the geometric support to encompass out-of-distribution high-reward regions. By exploring this extended geometry, ME-AM identifies robust actions while preserving the absolute continuity of the generative vector field. Empirically, ME-AM demonstrates competitive or superior performance compared to prior state-of-the-art (SOTA) methods across a diverse suite of sparse-reward continuous control environments.

Related papers

• ISEP: Implicit Support Expansion for Offline Reinforcement Learning via Stochastic Policy Optimization [6.976223727680188]
  Implicit Support Expansion via Policy optimization (ISEP)<n>ISEP uses a value function between in-distribution data and policy samples to implicitly expand the feasible action support.<n> Conditional Flow Matching.
  arXiv  Detail & Related papers  (2026-05-18T12:39:30Z)
• Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling [3.6266846456338695]
  Maximum entropy reinforcement learning (MaxEnt RL) has become a standard framework for sequential decision making.<n>We propose Truncated Rectified Flow Policy, a framework built on a hybrid deterministic-stochastic architecture.
  arXiv  Detail & Related papers  (2026-04-10T09:44:28Z)
• Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching [42.26405543045843]
  Action-based Policy Compression (APC) is widely recognized as sample-inefficient.<n>We introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage.<n>These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors.
  arXiv  Detail & Related papers  (2026-03-27T23:16:27Z)
• From Sparse to Dense: Multi-View GRPO for Flow Models via Augmented Condition Space [78.36537400975298]
  Group Relative Policy Optimization (GRPO) has emerged as a powerful framework for preference alignment in text-to-image (T2I) flow models.<n>We propose Multi-View GRPO, a novel approach that enhances relationship exploration by augmenting the condition space.<n>MV-GRPO achieves superior alignment performance over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-03-13T04:35:13Z)
• Modality-Guided Mixture of Graph Experts with Entropy-Triggered Routing for Multimodal Recommendation [6.815406906400871]
  Multimodal recommendation enhances ranking by integrating user-item interactions with item content.<n>Existing approaches often rely on shared fusion pathways, leading to entangled representations and modality imbalance.<n>We propose a Modality-Guided Mixture of Adaptive Graph Experts Network with Progressive Entropy-Triggered Routing for Multimodal Recommendation.
  arXiv  Detail & Related papers  (2026-02-24T09:36:45Z)
• Mean-Field Path-Integral Diffusion: From Samples to Interacting Agents [0.0]
  Mean-Field Path-Integral Diffusion (MF-PID) is a framework in which samples are promoted to interacting agents whose drift depends self-consistently on the evolving population density.<n>MF-PID achieves 19--24% reductions in cumulative control energy over independent-agent baselines while matching the prescribed terminal distribution exactly.
  arXiv  Detail & Related papers  (2026-02-23T21:14:06Z)
• The Offline-Frontier Shift: Diagnosing Distributional Limits in Generative Multi-Objective Optimization [56.39938641873341]
  We show that generative methods systematically underperform evolutionary alternatives with respect to other metrics, such as generational distance.<n>We argue that overcoming this limitation requires out-of-distribution sampling in objective space.<n>Our results position offline MOO as a distribution-shift--limited problem and provide a diagnostic lens for understanding when and why generative optimization methods fail.
  arXiv  Detail & Related papers  (2026-02-11T18:38:40Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• Denoising and Alignment: Rethinking Domain Generalization for Multimodal Face Anti-Spoofing [47.24147617685829]
  Face Anti-Spoofing (FAS) is essential for the security of facial recognition systems in diverse scenarios.<n>We introduce the textbfMultitextbfmodal textbfDenoising and textbfAlignment (textbfMMDA) framework.<n>By leveraging the zero-shot generalization capability of CLIP, the MMDA framework effectively suppresses noise in multimodal data.
  arXiv  Detail & Related papers  (2025-05-14T15:36:44Z)
• Offline Multi-agent Reinforcement Learning via Score Decomposition [51.23590397383217]
  offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts.<n>This work is the first work to explicitly address the distributional gap between offline and online MARL.
  arXiv  Detail & Related papers  (2025-05-09T11:42:31Z)
• Generalization Bounds of Surrogate Policies for Combinatorial Optimization Problems [53.03951222945921]
  We analyze smoothed (perturbed) policies, adding controlled random perturbations to the direction used by the linear oracle.<n>Our main contribution is a generalization bound that decomposes the excess risk into perturbation bias, statistical estimation error, and optimization error.<n>We illustrate the scope of the results on applications such as vehicle scheduling, highlighting how smoothing enables both tractable training and controlled generalization.
  arXiv  Detail & Related papers  (2024-07-24T12:00:30Z)
• Unified framework for continuity of sandwiched Rényi divergences [0.0]
  We prove continuity bounds for entropic quantities related to sandwiched R'enyi divergences.<n>In a separate contribution, we use the ALAFF method to study the stability of approximate quantum Markov chains.
  arXiv  Detail & Related papers  (2023-08-23T21:09:54Z)
• Iterative Amortized Policy Optimization [147.63129234446197]
  Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control. From the variational inference perspective, policy networks are a form of textitamortized optimization, optimizing network parameters rather than the policy distributions directly. We demonstrate that iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.
  arXiv  Detail & Related papers  (2020-10-20T23:25:42Z)
• Implicit Distributional Reinforcement Learning [61.166030238490634]
  implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs) Semi-implicit actor (SIA) powered by a flexible policy distribution. We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
  arXiv  Detail & Related papers  (2020-07-13T02:52:18Z)
• A maximum-entropy approach to off-policy evaluation in average-reward MDPs [54.967872716145656]
  This work focuses on off-policy evaluation (OPE) with function approximation in infinite-horizon undiscounted Markov decision processes (MDPs) We provide the first finite-sample OPE error bound, extending existing results beyond the episodic and discounted cases. We show that this results in an exponential-family distribution whose sufficient statistics are the features, paralleling maximum-entropy approaches in supervised learning.
  arXiv  Detail & Related papers  (2020-06-17T18:13:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.