Deep Pre-Alignment for VLMs

• URL: http://arxiv.org/abs/2605.15300v1
• Date: Thu, 14 May 2026 18:14:15 GMT
• Title: Deep Pre-Alignment for VLMs
• Abstract summary: Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector.<n>We propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver.
• Score: 18.93949300827106
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9\% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.

Related papers

• Unlocking Dense Metric Depth Estimation in VLMs [36.431792667223796]
  Vision-Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding.<n>We propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor.
  arXiv  Detail & Related papers  (2026-05-15T11:54:17Z)
• From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion [91.35078719566472]
  Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection.<n>We introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities.
  arXiv  Detail & Related papers  (2026-01-15T18:59:10Z)
• LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs [52.24096832965001]
  We present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method.<n>The PVC method can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding.<n>Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x.
  arXiv  Detail & Related papers  (2025-11-26T08:11:10Z)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models [21.197083685420584]
  We show the role of the projector in compressing vision embeddings and aligning them with word embeddings.<n>We then examine patch-level alignment -- the alignment between each vision patch and its corresponding semantic words.<n>Our experiments show that patch-aligned training achieves stronger compression capability and improved patch-level alignment.
  arXiv  Detail & Related papers  (2025-05-22T22:10:27Z)
• LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation [52.58791563814837]
  Large foundation models trained on large-scale vision-language data can boost Open-Vocabulary Object Detection (OVD)<n>This paper presents a systematic method to enhance visual grounding by utilizing decoder layers of the Large Language Models (LLMs)<n>We find that intermediate LLM layers already encode rich spatial semantics; adapting only the early layers yields most of the gain.
  arXiv  Detail & Related papers  (2025-03-18T00:50:40Z)
• LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer [110.39467860530819]
  Vision transformers (ViTs) are widely employed in multimodal large language models (MLLMs) for visual encoding.<n>We present LLaVA-UHD v2, an MLLM with advanced perception abilities by introducing a well-designed vision-language projector.<n>Hiwin transformer enhances MLLM's ability to capture diverse multi-modal visual granularities, by incorporating our constructed high-resolution semantic pyramid.
  arXiv  Detail & Related papers  (2024-12-18T14:07:46Z)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders [34.421335513040795]
  Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. We introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder.
  arXiv  Detail & Related papers  (2024-04-09T02:51:05Z)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models [73.40350756742231]
  Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored.
  arXiv  Detail & Related papers  (2024-02-12T18:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.