PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

• URL: http://arxiv.org/abs/2606.19534v1
• Date: Wed, 17 Jun 2026 19:27:55 GMT
• Title: PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models
• Abstract summary: PerceptionDLM is a multimodal diffusion language model optimized for efficient parallel region perception.<n>We introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions.<n>Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception.
• Score: 45.78343775270807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we propose PerceptionDLM, a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, a strong foundational baseline that achieves state-of-the-art performance among open-source diffusion MLLMs, our architecture fully leverages the parallel decoding nature of DLMs. Specifically, we introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To systematically evaluate the parallelism property of visual perception capability for DLMs, we construct a new Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference efficiency. Experiments demonstrate that PerceptionDLM maintains competitive performance in region captioning while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception. To the best of our knowledge, we are the first to achieve parallel region caption and perception by leveraging the advantages of diffusion language models. Code, models, and datasets are released.

Related papers

• PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs [59.78917775399492]
  Multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability.<n>We propose a training-free framework to mitigate this degradation.
  arXiv  Detail & Related papers  (2026-01-12T15:27:51Z)
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints [100.02131897927484]
  This paper focuses on the native training of Multimodal Large Language Models (MLLMs) in an end-to-end manner.<n>We propose a native MLLM called NaViL, combined with a simple and cost-effective recipe.<n> Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs.
  arXiv  Detail & Related papers  (2025-10-09T17:59:37Z)
• Beyond CNNs: Efficient Fine-Tuning of Multi-Modal LLMs for Object Detection on Low-Data Regimes [0.0]
  We compare fine-tuned traditional CNNs, zero-shot pre-trained multi-modal LLMs, and fine-tuned multi-modal LLMs on the challenging task of artificial text overlay detection in images.<n>A key contribution of our study is demonstrating that LLMs can be effectively fine-tuned on very limited data to achieve up to 36% accuracy improvement.<n>Our work contributes to the broader effort of bridging vision and language, offering novel insights into efficient cross-modal learning strategies.
  arXiv  Detail & Related papers  (2025-10-03T18:53:18Z)
• ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way [8.242020781632801]
  ByDeWay is a training-free framework designed to enhance the performance of Multimodal Large Language Models (MLLMs)<n>ByDeWay uses a novel prompting strategy called Layered-Depth-Based Prompting (LDP)<n>It segments the scene into closest, mid-range, and farthest layers using monocular depth estimation, then generates region-specific captions with a grounded vision-language model.
  arXiv  Detail & Related papers  (2025-07-11T15:21:49Z)
• Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos [53.723410664944566]
  We present Perceive Anything Model (PAM), a framework for comprehensive region-level visual understanding in images and videos.<n>Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation.<n>A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features into multi-modal tokens.
  arXiv  Detail & Related papers  (2025-06-05T17:51:39Z)
• TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models [23.916205754112774]
  Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks.<n>We propose TAMP, a simple yet effective pruning framework tailored for MLLMs.<n>We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities.
  arXiv  Detail & Related papers  (2025-04-14T05:44:38Z)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception [63.03288425612792]
  We propose bfAnyRef, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references. Our model achieves state-of-the-art results across multiple benchmarks, including diverse modality referring segmentation and region-level referring expression generation.
  arXiv  Detail & Related papers  (2024-03-05T13:45:46Z)
• ModaVerse: Efficiently Transforming Modalities with LLMs [25.49713745405194]
  We introduce ModaVerse, a Multi-modal Large Language Model capable of comprehending and transforming content across various modalities. We propose a novel Input/Output (I/O) alignment mechanism that operates directly at the level of natural language.
  arXiv  Detail & Related papers  (2024-01-12T06:28:54Z)
• InfMLLM: A Unified Framework for Visual-Language Tasks [44.29407348046122]
  multimodal large language models (MLLMs) have attracted growing interest. This work delves into enabling LLMs to tackle more vision-language-related tasks. InfMLLM achieves either state-of-the-art (SOTA) performance or performance comparable to recent MLLMs.
  arXiv  Detail & Related papers  (2023-11-12T09:58:16Z)
• Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models [50.07056960586183]
  We propose Position-enhanced Visual Instruction Tuning (PVIT) to extend the functionality of Multimodal Large Language Models (MLLMs) This integration promotes a more detailed comprehension of images for the MLLM. We present both quantitative experiments and qualitative analysis that demonstrate the superiority of the proposed model.
  arXiv  Detail & Related papers  (2023-08-25T15:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.