Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

• URL: http://arxiv.org/abs/2604.11510v1
• Date: Mon, 13 Apr 2026 14:13:06 GMT
• Title: Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization
• Abstract summary: Policy Split is a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt.<n>While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives.
• Score: 53.051213470905935
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.

Related papers

• Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling [31.631345414385382]
  Multimodal motion forecasting is inherently under-supervised.<n>We propose Mode-as-Sequence, a unified decoding framework that translates an unordered mode set into an ordered mode sequence.<n>We show consistent improvements in both ranking-oriented metrics and best-of-K accuracy across datasets, horizons, and object types.
  arXiv  Detail & Related papers  (2026-05-21T11:37:17Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models [45.592930731636386]
  We propose $M2$-REPA, the first representation alignment method tailored for multi-modal video generation.<n>We first decouple modality-specific features from the diffusion model's intermediate representations, then align each with its corresponding expert foundation model.<n>Our method significantly outperforms baselines in visual quality and long-term consistency.
  arXiv  Detail & Related papers  (2026-05-03T14:22:22Z)
• OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks [81.07598709704628]
  We present a novel RL training objective that replaces standard linear scaling with non-linear distributional matching.<n>We also present OpenVLThinkerV2, a highly robust, general-purpose multimodal model.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• Primary-Fine Decoupling for Action Generation in Robotic Imitation [91.2899765310853]
  Multi-modal distribution in robotic manipulation action sequences poses critical challenges for imitation learning.<n>We propose Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations.<n>PF-DAG outperforms state-of-the-art baselines across 56 tasks from Adroit, DexArt, and MetaWorld benchmarks.
  arXiv  Detail & Related papers  (2026-02-25T08:36:45Z)
• Dual-level Modality Debiasing Learning for Unsupervised Visible-Infrared Person Re-Identification [59.59359638389348]
  We propose a Dual-level Modality Debiasing Learning framework that implements debiasing at both the model and optimization levels.<n>Experiments on benchmark datasets demonstrate that DMDL could enable modality-invariant feature learning and a more generalized model.
  arXiv  Detail & Related papers  (2025-12-03T12:43:16Z)
• Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning [43.07723859242318]
  We propose a novel adaptive reasoning paradigm, Mixture-of-Visual-Thoughts (MoVT)<n>MoVT unifies different reasoning modes within a single model and guides it to select the appropriate mode based on context.<n>AdaVaR effectively guides the model to learn and differentiate multiple modes and perform context-adaptive mode selection.
  arXiv  Detail & Related papers  (2025-09-26T04:33:53Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [57.577843653775]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning [9.38848713730931]
  offline reinforcement learning seeks to learn optimal policies from static datasets without interacting with the environment.<n>Existing methods often assume unimodal behaviour policies, leading to suboptimal performance when this assumption is violated.<n>We propose weighted imitation Learning on One Mode (LOM), a novel approach that focuses on learning from a single, promising mode of the behaviour policy.
  arXiv  Detail & Related papers  (2024-12-04T11:57:36Z)
• Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient [26.675822002049372]
  Deep Diffusion Policy Gradient (DDiffPG) is a novel actor-critic algorithm that learns from scratch multimodal policies. DDiffPG forms a multimodal training batch and utilizes mode-specific Q-learning to mitigate the inherent greediness of the RL objective. Our approach further allows the policy to be conditioned on mode-specific embeddings to explicitly control the learned modes.
  arXiv  Detail & Related papers  (2024-06-02T09:32:28Z)
• PSNet: Parallel Symmetric Network for Video Salient Object Detection [85.94443548452729]
  We propose a VSOD network with up and down parallel symmetry, named PSNet. Two parallel branches with different dominant modalities are set to achieve complete video saliency decoding.
  arXiv  Detail & Related papers  (2022-10-12T04:11:48Z)
• PARAFAC2 AO-ADMM: Constraints in all modes [6.901159341430921]
  We propose an alternating direction method of multipliers (ADMM)-based algorithm for fitting PARAFAC2 and widen the possible regularisation penalties to any proximable function. Our numerical experiments demonstrate that the proposed ADMM-based approach for PARAFAC2 can accurately recover the underlying components from simulated data.
  arXiv  Detail & Related papers  (2021-02-03T14:42:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.