DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Models

• URL: http://arxiv.org/abs/2605.08902v1
• Date: Sat, 09 May 2026 11:47:43 GMT
• Title: DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Models
• Abstract summary: Existing methods overlook the inherent and dynamic differences in information density and semantic scope between text tags and image blocks.<n>This paper proposes a novel framework for dynamic cross-modal alignment with continuous detail introduction.
• Score: 7.835293178197385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, pre-trained visual-linguistic models have demonstrated tremendous potential, becoming a crucial foundational framework for numerous downstream tasks. However, the information density between text and images is not uniformly distributed. Existing methods often overlook the inherent and dynamic differences in information density and semantic scope between text tags and image blocks. These common uniform alignment strategies result in coarse-grained cross-modal interactions and loss of fine semantic details. Moreover, pursuing finer alignment typically requires substantial computational overhead, limiting practical model deployment. To address this challenge, this paper proposes a novel framework for dynamic cross-modal alignment with continuous detail introduction. First, we design a dynamically adaptive cross-modal matching mechanism that uses a learnable matching function to dynamically assign varying numbers and sizes of image tags to text tags of the same size but different information density, enabling more precise attention interaction. Second, we develop a continuous detail introduction module to progressively incorporate high-resolution visual feature enhancement into the alignment process. Extensive experiments across multiple benchmarks demonstrate significant improvements in the accuracy of various downstream tasks while reducing computational overhead.

Related papers

• ViQ: Text-Aligned Visual Quantized Representations at Any Resolution [91.46185855575789]
  We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations.<n>Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization.<n>Experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders.
  arXiv  Detail & Related papers  (2026-06-25T17:29:27Z)
• Multimodal Large Language Models for Multi-Subject In-Context Image Generation [56.20395856287325]
  Music is the first MLLM specifically designed for textbfMUlti-textbfSubject textbfIn-textbfContext image generation.<n>We enhance the model's understanding of multi-subject semantic relationships through a vision chain-of-thought mechanism.<n>By incorporating complex subject images during training, we improve the model's capacity for chained reasoning.
  arXiv  Detail & Related papers  (2026-04-08T15:37:42Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception [54.53657134205492]
  UniAlignment is a unified multimodal generation framework within a single diffusion transformer.<n>It incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness.<n>We present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions.
  arXiv  Detail & Related papers  (2025-09-28T09:11:30Z)
• Infusing fine-grained visual knowledge to Vision-Language Models [5.487134463783365]
  Large-scale contrastive pre-training produces powerful Vision-and-Language Models (VLMs)<n>We propose a fine-tuning method explicitly designed to achieve optimal balance between fine-grained domain adaptation and retention of the pretrained VLM's broad multimodal knowledge.<n>Our approach consistently achieves strong results, notably retaining the visual-text alignment without utilizing any text data or the original text encoder during fine-tuning.
  arXiv  Detail & Related papers  (2025-08-16T19:12:09Z)
• Language Guided Domain Generalized Medical Image Segmentation [68.93124785575739]
  Single source domain generalization holds promise for more reliable and consistent image segmentation across real-world clinical settings. We propose an approach that explicitly leverages textual information by incorporating a contrastive learning mechanism guided by the text encoder features. Our approach achieves favorable performance against existing methods in literature.
  arXiv  Detail & Related papers  (2024-04-01T17:48:15Z)
• Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking [0.5242869847419834]
  We propose a Dynamic Visual Semantic Sub-Embeddings framework (DVSE) to reduce the information entropy. To encourage the generated candidate embeddings to capture various semantic variations, we construct a mixed distribution. We compare the performance with existing set-based method using four image feature encoders and two text feature encoders on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-09-15T04:39:11Z)
• Alignment-free HDR Deghosting with Semantics Consistent Transformer [76.91669741684173]
  High dynamic range imaging aims to retrieve information from multiple low-dynamic range inputs to generate realistic output. Existing methods often focus on the spatial misalignment across input frames caused by the foreground and/or camera motion. We propose a novel alignment-free network with a Semantics Consistent Transformer (SCTNet) with both spatial and channel attention modules.
  arXiv  Detail & Related papers  (2023-05-29T15:03:23Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)
• DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation [7.781425222538382]
  DiverGAN is a framework to generate diverse, plausible and semantically consistent images according to a natural-language description. DiverGAN adopts two novel word-level attention modules, i.e., a channel-attention module (CAM) and a pixel-attention module (PAM) Conditional Adaptive Instance-Layer Normalization (CAdaILN) is introduced to enable the linguistic cues from the sentence embedding to flexibly manipulate the amount of change in shape and texture.
  arXiv  Detail & Related papers  (2021-11-17T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.