Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Model

• URL: http://arxiv.org/abs/2606.14792v1
• Date: Thu, 11 Jun 2026 07:33:46 GMT
• Title: Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Model
• Abstract summary: multimodal discrete diffusion models are effective alternatives to AR models for reinforcement learning in interleaved reasoning.<n>Joint reward assignment, which employs a shared reward signal across modalities, introduces cross-modal interference during RL updates.<n>We propose factorized reward assignment, a strategy that assigns rewards independently to text and vision segments.
• Score: 70.56994065819471
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: RL-based post-training has been widely adopted to enable interleaved visual and textual reasoning in unified multimodal models capable of both text and image generation. However, most existing approaches are built upon autoregressive (AR) unified models, which require full image regeneration during visual reasoning. In this work, we demonstrate that multimodal discrete diffusion models are effective alternatives to AR models for reinforcement learning in interleaved reasoning, owing to their ability to perform efficient visual rollouts via localized visual editing rather than full image-token regeneration. This reduces rollout computation during GRPO by 26.9\% compared to AR baselines, with minimal performance drop. Despite the improved efficiency, we find that joint reward assignment, which employs a shared reward signal across modalities, introduces cross-modal interference between unrelated image and text token sequences during RL updates. To address this issue, we propose factorized reward assignment, a strategy that assigns rewards independently to text and vision segments. With factorized reward assignment, our RL approach achieves an 11.2% improvement over joint reward assignment and a 38.04% improvement over the base model.

Related papers

• Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation [81.40978077888693]
  Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance.<n>Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens.<n>We integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations.
  arXiv  Detail & Related papers  (2026-03-05T04:45:49Z)
• Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward [93.04811239892852]
  Reinforcement Learning (RL) has recently been incorporated into diffusion models.<n>In this paper, we investigate how to effectively integrate RL into diffusion-based restoration models.
  arXiv  Detail & Related papers  (2025-11-03T14:57:57Z)
• AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning [56.71089466532673]
  We propose AR-GRPO, an approach to integrate online RL training into autoregressive (AR) image generation models.<n>We conduct comprehensive experiments on both class-conditional (i.e., class-to-image) and text-conditional (i.e., text-to-image) image generation tasks.<n>Our results show consistent improvements across various evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-09T10:37:26Z)
• PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning [50.21619363035618]
  We propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks.<n>We introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional diversity.<n>Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin.
  arXiv  Detail & Related papers  (2025-06-17T18:25:56Z)
• Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization [1.1510009152620668]
  Fine-tuning pre-trained generative models with Reinforcement Learning (RL) has emerged as an effective approach for aligning outputs with human preferences.<n>We show that RL-based fine-tuning is both efficient and effective for VAR models, benefiting particularly from their fast inference speeds.
  arXiv  Detail & Related papers  (2025-05-29T10:45:38Z)
• Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO [68.44918104224818]
  Autoregressive image generation presents unique challenges distinct from Chain-of-Thought (CoT) reasoning.<n>This study provides the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation.<n>Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities potentially enhance the generalization potential of the applied RL algorithms.
  arXiv  Detail & Related papers  (2025-05-22T17:59:49Z)
• RL for Consistency Models: Faster Reward Guided Text-to-Image Generation [15.238373471473645]
  We propose a framework for fine-tuning consistency models viaReinforcement Learning (RL) Our framework, called Reinforcement Learning for Consistency Model (RLCM), frames the iterative inference process of a consistency model as an RL procedure. Comparing to RL finetuned diffusion models, RLCM trains significantly faster, improves the quality of the generation measured under the reward objectives, and speeds up the inference procedure by generating high quality images with as few as two inference steps.
  arXiv  Detail & Related papers  (2024-03-25T15:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.