Dual-Flow Reinforcement Learning with State-Aware Exploration

• URL: http://arxiv.org/abs/2606.29820v1
• Date: Mon, 29 Jun 2026 05:57:33 GMT
• Title: Dual-Flow Reinforcement Learning with State-Aware Exploration
• Abstract summary: In continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions.<n>We propose Dual-Flow RL, a unified actor-critic framework that jointly models a continuous return distribution and a multimodal policy distribution.<n>Experiments on DeepMind Control Suite and Humanoid-Bench show that Dual-Flow RL achieves state-of-the-art performance on most tasks.
• Score: 16.944702024760094
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challenging. Existing value estimation methods using unimodal Gaussians restrict expressiveness and yield biased estimates. Recent generative policies can represent multimodal actions but often collapse to a few modes and under-explore high-value areas of the action space. Motivated by these challenges, we propose Dual-Flow RL, a unified actor-critic framework that jointly models a continuous return distribution and a multimodal policy distribution using conditional flow matching (CFM). This design supports reliable value estimation and sustained multimodal exploration. To further enhance exploration, we introduce an Entropy-Covariance Exploration Regulator (ECER) that enables state-aware exploration regulation leveraging policy entropy and action-uncertainty covariance. Experiments on DeepMind Control Suite and Humanoid-Bench show that Dual-Flow RL achieves state-of-the-art performance on most tasks, significantly outperforming prior diffusion-based and flow-based methods.

Related papers

• FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning [63.80677039120727]
  FlowR2A unifies the dense supervision of scoring-based methods with the proposal generation of anchor-based methods in a single generative model.<n>To balance hard safety constraints against soft progress objectives, we introduce fine-grained per-timestep reward conditioning and reward noise augmentation.<n>The generative formulation naturally supports controllable test-time sampling via reward guidance and anchored sampling, producing high-quality proposals.
  arXiv  Detail & Related papers  (2026-06-23T07:21:36Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks [81.07598709704628]
  We present a novel RL training objective that replaces standard linear scaling with non-linear distributional matching.<n>We also present OpenVLThinkerV2, a highly robust, general-purpose multimodal model.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• From Flow to One Step: Real-Time Multi-Modal Trajectory Policies via Implicit Maximum Likelihood Estimation-based Distribution Distillation [18.70033095161235]
  We propose a framework that distills a Conditional Flow Matching expert into a fast single-step student via Implicit Likelihood Estimation (IMLE)<n>A bi-directional Chamfer distance provides a set-level objective that promotes both mode coverage and fidelity.<n>A unified perception encoder further integrates multi-view RGB, depth, point clouds, and proprioception into a geometry-aware representation.
  arXiv  Detail & Related papers  (2026-03-10T09:30:05Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks [0.3425341633647625]
  Tasks involving high-risk-high-return actions often exhibit multimodal action distributions and returns.<n>Most reinforcement learning methods assume unimodal Gaussian policies and rely on scalar-valued critics.<n>We propose a reinforcement learning framework that discretizes continuous action spaces to approximate multimodal distributions.
  arXiv  Detail & Related papers  (2025-10-20T06:54:53Z)
• G$^2$RPO: Granular GRPO for Precise Reward in Flow Models [74.21206048155669]
  We propose a novel Granular-GRPO (G$2$RPO) framework that achieves precise and comprehensive reward assessments of sampling directions.<n>We introduce a Multi-Granularity Advantage Integration module that aggregates advantages computed at multiple diffusion scales.<n>Our G$2$RPO significantly outperforms existing flow-based GRPO baselines.
  arXiv  Detail & Related papers  (2025-10-02T12:57:12Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [57.577843653775]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• Breaking Imitation Bottlenecks: Reinforced Diffusion Powers Diverse Trajectory Generation [20.106116218594266]
  DIVER is an end-to-end autonomous driving framework that integrates reinforcement learning and diffusion-based generation.<n>We show that DIVER significantly improves trajectory diversity, effectively addressing the mode collapse problem inherent in imitation learning.
  arXiv  Detail & Related papers  (2025-07-05T14:19:19Z)
• Distributional Soft Actor-Critic with Diffusion Policy [12.762838783617658]
  This paper proposes a distributional reinforcement learning algorithm called DSAC-D (Distributed Soft Actor Critic with Policy Diffusion) to address the challenges of estimating bias in value functions.<n>The proposed algorithm achieves state-of-the-art (SOTA) performance in all 9 control tasks, with significant suppression of estimation bias and total average return improvement of over 10% compared to existing mainstream algorithms.
  arXiv  Detail & Related papers  (2025-07-02T05:50:10Z)
• Continual Multimodal Contrastive Learning [99.53621521696051]
  Multimodal Contrastive Learning (MCL) advances in aligning different modalities and generating multimodal representations in a joint space.<n>However, a critical yet often overlooked challenge remains: multimodal data is rarely collected in a single process, and training from scratch is computationally expensive.<n>In this paper, we formulate CMCL through two specialized principles of stability and plasticity.<n>We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient is prevented from interfering with the previously learned knowledge.
  arXiv  Detail & Related papers  (2025-03-19T07:57:08Z)
• Generalizing Multimodal Variational Methods to Sets [35.69942798534849]
  This paper presents a novel variational method on sets called the Set Multimodal VAE (SMVAE) for learning a multimodal latent space. By modeling the joint-modality posterior distribution directly, the proposed SMVAE learns to exchange information between multiple modalities and compensate for the drawbacks caused by factorization.
  arXiv  Detail & Related papers  (2022-12-19T23:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.