Multi-Branch Policy Optimization for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2608.07581v1
• Date: Wed, 05 Aug 2026 04:00:09 GMT
• Title: Multi-Branch Policy Optimization for Multimodal Large Language Models
• Abstract summary: Group-based reinforcement learning methods for multimodal large language models rely on trajectory-level credit assignment.<n> multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings.<n>We propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries.
• Score: 21.856599797665833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.

Related papers

• Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process [47.3257468640999]
  We introduce textbfBRAID (textbfBridging intetextbfRletextbfAved multtextbfI-modal reasoning as a unified textbfDecision process)<n>BRAID computes a shared trajectory-level advantage and propagates it coherently into both text tokens and image denoising paths.<n> Experiments on spatial reasoning and visual perception benchmarks show that BRAID consistently outperforms various baselines.
  arXiv  Detail & Related papers  (2026-07-04T07:40:43Z)
• Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization [89.68681087743876]
  Multimodal Large Language Models (MLLMs) have incentivized models to think with images'' by actively invoking visual tools during multi-turn reasoning.<n>The common Reinforcement Learning practice of relying on outcome-based rewards ignores the fact that textual plausibility often masks executive failure.<n>We introduce Multimodal Agentic Policy Optimization (MAPO), bridging the gap between textual reasoning and visual actions generated by models within their Multimodal Chain-of-Thought.
  arXiv  Detail & Related papers  (2026-04-08T07:48:07Z)
• PLUME: Latent Reasoning Based Universal Multimodal Embedding [52.35354073629127]
  Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
  arXiv  Detail & Related papers  (2026-04-02T14:04:53Z)
• Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models [38.47027398567909]
  Perception-Grounded Policy Optimization (PGPO) is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level.<n>We show that PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors.<n>Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2026-04-02T09:53:20Z)
• LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation [51.071351994330605]
  Unified multimodal pretraining has emerged as a promising paradigm for jointly modeling language and vision within a single foundation model.<n>Existing approaches largely rely on implicit or indirect alignment signals and remain suboptimal for simultaneously supporting multimodal understanding and generation.<n>We propose LVRPO, a language-visual reinforcement-based preference optimization framework that explicitly aligns language and visual representations.
  arXiv  Detail & Related papers  (2026-03-29T13:38:21Z)
• Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought [73.39221516441624]
  Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories.<n>Existing Reinforcement Learning with Verifiable Rewards (RLVR) methods treat CoT uniformly without distinguishing varying degrees of visual grounding.<n>We propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior to hidden state similarity and integrates it with token entropy.
  arXiv  Detail & Related papers  (2026-03-24T06:38:00Z)
• Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models [67.45032003041399]
  We propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs.<n>MPCO adaptively balances the importance of different paradigm representations and guides the global optimisation.<n>Our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs.
  arXiv  Detail & Related papers  (2026-03-05T06:01:26Z)
• Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization [68.64764778089229]
  We propose MISP-DPO, the first framework to incorporate multiple, semantically diverse negative images in multimodal DPO.<n>Our method embeds prompts and candidate images in CLIP space and applies a sparse autoencoder to uncover semantic deviations into interpretable factors.<n>Experiments across five benchmarks demonstrate that MISP-DPO consistently improves multimodal alignment over prior methods.
  arXiv  Detail & Related papers  (2025-09-30T03:24:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.