FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance

• URL: http://arxiv.org/abs/2605.30749v1
• Date: Fri, 29 May 2026 02:25:03 GMT
• Title: FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
• Abstract summary: We introduce FLAG (textbfFlow policy with textbfLatent-textbfAugmented textbfGuidance)<n>FLAG enables expressive policy optimization with limited importance samples and scales to high-dimensional control tasks.
• Score: 20.708304655938438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Maximum entropy reinforcement learning (MaxEnt-RL) enables robust exploration, yet practical implementations often restrict policies to simple Gaussians. While recent approaches incorporate expressive generative policies via importance-weighted supervised learning, they are prone to importance weight collapse, which limits their scalability in high-dimensional action spaces. Our key insight is to mitigate this limitation by localizing the sampling region, avoiding the weight degeneracy induced by importance sampling over the entire action space. To instantiate this insight, we introduce \textbf{FLAG} (\textbf{F}low policy with \textbf{L}atent-\textbf{A}ugmented \textbf{G}uidance). FLAG augments the state space with a flow latent variable and optimizes a provably consistent proxy MaxEnt-RL objective. We empirically demonstrate that FLAG enables expressive policy optimization with limited importance samples and scales to high-dimensional control tasks. Furthermore, FLAG achieves state-of-the-art performance across challenging benchmarks. Our project webpage: https://flag-rl.github.io/

Related papers

• Revisiting Topological Graphs for Macro Action based Closed-loop Reinforcement Learning of Vision Language Navigation in Continuous Environment [24.27274618177781]
  Vision-Language Navigation in Continuous Environments (VLN-CE) requires an agent to follow natural language instructions through unseen environments.<n>We reformulate VLN-CE as a Hierarchical Markov Decision Process (MDP), explicitly decoupling high-level planning from low-level control.<n>Our model achieves state-of-the-art performance on the R2R-CE and RxR-CE benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T14:24:29Z)
• Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR [53.27792011950384]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models.<n>We propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR.<n>Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts.
  arXiv  Detail & Related papers  (2026-05-15T08:22:59Z)
• LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning [90.86828952599147]
  We present LaST-R1, a novel reinforcement learning framework designed to harness "latent reasoning-before-acting" policies.<n>LaST-R1 achieves a near-perfect 99.9% average success rate on the LIBERO benchmark.<n>In real-world deployments, LaST-R1 yields up to a 22.5% average improvement over SOTA supervised fine-tuning approach.
  arXiv  Detail & Related papers  (2026-04-30T17:59:52Z)
• Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching [8.665369041430969]
  Flow Matching (FM) enables one-step generation, but integrating it into Entropy Reinforcement Learning (MaxEnt RL) is challenging.<n>We propose textbfFlow-based textbfLog-likelihood-textbfAware textbfMaximum textbfEntropy RL (textbfFLAME), a principled framework that addresses these challenges.
  arXiv  Detail & Related papers  (2026-02-02T03:54:11Z)
• Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization [1.974921946982281]
  Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs)<n>Recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold.<n>We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs.
  arXiv  Detail & Related papers  (2026-01-30T05:38:44Z)
• The Flexibility Trap: Why Arbitrary Order Limits Reasoning Potential in Diffusion Language Models [67.58848748317506]
  Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs.<n>In this paper, we reveal a counter-intuitive reality: arbitrary order generation, in its current form, narrows rather than expands the reasoning boundary of dLLMs.
  arXiv  Detail & Related papers  (2026-01-21T16:41:58Z)
• Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models [53.339700196282905]
  A key challenge in applying reinforcement learning to large language models (dLLMs) is the intractability of their likelihood functions.<n>We propose a memory-efficient RL algorithm that maximizes a specially constructed lower bound of the ELBO-based objective.<n> Experiments show that BGPO significantly outperforms previous RL algorithms for dLLMs in math problem solving, code generation, and planning tasks.
  arXiv  Detail & Related papers  (2025-10-13T17:47:50Z)
• Massively Scaling Explicit Policy-conditioned Value Functions [16.387595437722613]
  We introduce a scaling strategy for Explicit Policy-Conditioned Value Functions (EPVFs)<n>EPVFs learn a value function V(theta) that is explicitly conditioned on the policy parameters, enabling direct gradient-based updates to the parameters of any policy.<n>We show that EPVFs can be scaled to solve complex tasks, such as a custom Ant environment, and can compete with state-of-the-art Deep Reinforcement Learning (DRL) baselines.
  arXiv  Detail & Related papers  (2025-02-17T16:02:54Z)
• Maximum Entropy Reinforcement Learning with Diffusion Policy [24.889485955864547]
  In this paper, we employ the diffusion model, a powerful generative model capable of capturing complex multimodal distributions, as the policy representation to fulfill the MaxEnt RL objective.<n>Our method enables efficient exploration and brings the policy closer to the optimal MaxEnt policy.
  arXiv  Detail & Related papers  (2025-02-17T09:55:58Z)
• Action abstractions for amortized sampling [49.384037138511246]
  We propose an approach to incorporate the discovery of action abstractions, or high-level actions, into the policy optimization process. Our approach involves iteratively extracting action subsequences commonly used across many high-reward trajectories and chunking' them into a single action that is added to the action space.
  arXiv  Detail & Related papers  (2024-10-19T19:22:50Z)
• Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuning [65.31677646659895]
  Large language models demonstrate impressive performance on downstream tasks, yet they require extensive resource consumption when fully fine-tuning all parameters.<n>We propose a framework to clearly define task-specific directions (TSDs) and explore their properties and practical utilization challenges.<n>We then introduce a novel approach, LoRA-Dash, which aims to maximize the impact of TSDs during the fine-tuning process.
  arXiv  Detail & Related papers  (2024-09-02T08:10:51Z)
• Reparameterized Policy Learning for Multimodal Trajectory Optimization [61.13228961771765]
  We investigate the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces. We propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. We present a practical model-based RL method, which leverages the multimodal policy parameterization and learned world model.
  arXiv  Detail & Related papers  (2023-07-20T09:05:46Z)
• Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning [54.378444600773875]
  We introduce Successor Feature Landmarks (SFL), a framework for exploring large, high-dimensional environments. SFL drives exploration by estimating state-novelty and enables high-level planning by abstracting the state-space as a non-parametric landmark-based graph. We show in our experiments on MiniGrid and ViZDoom that SFL enables efficient exploration of large, high-dimensional state spaces.
  arXiv  Detail & Related papers  (2021-11-18T18:36:05Z)
• Deep Reinforcement Learning with Robust and Smooth Policy [90.78795857181727]
  We propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- textbfSmooth textbfRegularized textbfReinforcement textbfLearning ($textbfSR2textbfL$), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy.
  arXiv  Detail & Related papers  (2020-03-21T00:10:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.