Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

• URL: http://arxiv.org/abs/2608.26580v1
• Date: Thu, 27 Aug 2026 03:46:27 GMT
• Title: Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models
• Abstract summary: Diffusion multimodal large language models (dMLLMs) have emerged as a new decoding paradigm for multimodal generation.<n>We propose VIG-Sampler, which prioritizes tokens based on their attention to image tokens.<n>Experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler.
• Score: 20.55764219638966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.

Related papers

• Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models [9.95016675032699]
  Diffusion-based multimodal large language models (dMLLMs) decode by iteratively predicting tokens at multiple masked positions in parallel.<n>Existing confidence-based decoding ranks masked positions independently and commits the top-K positions, largely ignoring whether the committed tokens provide complementary visual grounding.<n>We propose Visual-Redundancy-Controlled Decoding (VRCD), a training-free inference-time decoding method that uses token-to-image attention to prioritize visually complementary positions.
  arXiv  Detail & Related papers  (2026-05-25T13:16:51Z)
• Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models [52.40742159500277]
  We show how to optimize Diffusion Multi-Modal Large Language Models (dMLLMs) through reinforcement learning (RL)<n>Our approach features a Sketch-Then-Paint training scheme that organizes updates into three distinct stages: global, structure, and refinement.<n> Experiments using two popular dMLLM backbones, MMaDA and Lumina-DiMOO, demonstrate substantial gains on the GenEval and DPG benchmarks.
  arXiv  Detail & Related papers  (2026-05-16T06:59:54Z)
• Bottleneck Tokens for Unified Multimodal Retrieval [16.707536543758344]
  Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps.<n>First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token as the sequence-level representation.<n>Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it.<n>We introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism.
  arXiv  Detail & Related papers  (2026-04-13T07:12:12Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• FoPru: Focal Pruning for Efficient Large Vision-Language Models [11.36025001578531]
  We propose Focal Pruning (FoPru), a training-free method that prunes visual tokens based on the attention-based token significance derived from the vision encoder. Our method can prune a large number of redundant tokens while maintaining high accuracy, leading to significant improvements in inference efficiency.
  arXiv  Detail & Related papers  (2024-11-21T14:22:38Z)
• FIRP: Faster LLM inference via future intermediate representation prediction [54.897493351694195]
  FIRP generates multiple tokens instead of one at each decoding step. We conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets.
  arXiv  Detail & Related papers  (2024-10-27T15:53:49Z)
• SEP: Self-Enhanced Prompt Tuning for Visual-Language Model [93.94454894142413]
  We introduce a novel approach named Self-Enhanced Prompt Tuning (SEP) SEP explicitly incorporates discriminative prior knowledge to enhance both textual-level and visual-level embeddings. Comprehensive evaluations across various benchmarks and tasks confirm SEP's efficacy in prompt tuning.
  arXiv  Detail & Related papers  (2024-05-24T13:35:56Z)
• Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration [54.897493351694195]
  We propose a novel parallel decoding approach, namely textithidden transfer, which decodes multiple successive tokens simultaneously in a single forward pass. In terms of acceleration metrics, we outperform all the single-model acceleration techniques, including Medusa and Self-Speculative decoding.
  arXiv  Detail & Related papers  (2024-04-18T09:17:06Z)
• Leveraging per Image-Token Consistency for Vision-Language Pre-training [52.825150269820696]
  Cross-modal masked language modeling (CMLM) is insufficient for vision-language pre-training. We propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training) The proposed EPIC method is easily combined with pre-training methods.
  arXiv  Detail & Related papers  (2022-11-20T12:10:53Z)
• Data Efficient Masked Language Modeling for Vision and Language [16.95631509102115]
  Masked language modeling (MLM) is one of the key sub-tasks in vision-language training. In the cross-modal setting, tokens in the sentence are masked at random, and the model predicts the masked tokens given the image and the text. We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings.
  arXiv  Detail & Related papers  (2021-09-05T11:27:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.