Bridging Hidden States in Vision-Language Models

• URL: http://arxiv.org/abs/2511.11526v1
• Date: Fri, 14 Nov 2025 17:55:25 GMT
• Title: Bridging Hidden States in Vision-Language Models
• Abstract summary: Vision-Language Models (VLMs) are a new family of models that align image content with natural language.<n>We propose a lightweight fusion module: a few cross-only, bidirectional attention layers placed near the top of both encoders.<n>The encoders remain non-causal and strong for understanding, while generation stays cleanly decoupled via an optional decoder.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are a new family of models that align image content with natural language. Existing approaches typically fuse either (a) early: by mixing tokens/features inside the encoders, or (b) late: by comparing pooled embeddings. Many methods also tie fusion to an autoregressive decoder. However, the hidden states of both modalities already carry rich, modality-specific structure (spatial layout in vision; syntax and semantics in text), so directly aligning these states is a natural way to match what the two modalities "think". We propose a lightweight fusion module: a few cross-only, bidirectional attention layers placed near the top of both encoders. Each layer projects the vision and text encoder hidden-state sequences into a shared space, attends across modalities, and sends gated residual updates back, with simple stabilizers to improve alignment. The encoders remain non-causal and strong for understanding, while generation stays cleanly decoupled via an optional decoder. Across standard retrieval, VQA, and visual reasoning benchmarks, BRIDGE outperforms comparable VLMs while preserving the bi-encoder efficiency of contrastive models. We make our code publicly available at https://github.com/jfeinashley/BRIDGE.

Related papers

• Stateful Visual Encoders for Vision-Language Models [85.81539882553659]
  Vision-language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes.<n>In existing open-weight VLMs, visual comparisons happen only inside the language model, while the visual encoder itself remains stateless.<n>We introduce a Stateful Visual, which conditions each visual representation on prior visual features.
  arXiv  Detail & Related papers  (2026-06-03T04:31:15Z)
• Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models [9.95016675032699]
  Diffusion-based multimodal large language models (dMLLMs) decode by iteratively predicting tokens at multiple masked positions in parallel.<n>Existing confidence-based decoding ranks masked positions independently and commits the top-K positions, largely ignoring whether the committed tokens provide complementary visual grounding.<n>We propose Visual-Redundancy-Controlled Decoding (VRCD), a training-free inference-time decoding method that uses token-to-image attention to prioritize visually complementary positions.
  arXiv  Detail & Related papers  (2026-05-25T13:16:51Z)
• A More Word-like Image Tokenization for MLLMs [26.120899392740203]
  We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch embeddings into coherent semantic units.<n>Across diverse multimodal benchmarks, DiVT matches or surpasses baselines with significantly fewer visual tokens.
  arXiv  Detail & Related papers  (2026-05-18T07:09:46Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• Decoupling Vision and Language: Codebook Anchored Visual Adaptation [20.393987361723724]
  Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning.<n>Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates.<n>We introduce CRAFT, a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space.
  arXiv  Detail & Related papers  (2026-02-23T02:39:26Z)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning [23.58934174168992]
  Autoregressive (AR) language models generate text one token at a time, which limits their inference speed.<n>We propose Convolutional decoding (Conv), a normalization-based method that narrows the decoding window without hard segmentation.<n>We also introduce Rejecting Rule-based Fine-Tuning (R2FT), a post-hoc training scheme that better aligns tokens at positions far from context.
  arXiv  Detail & Related papers  (2025-09-18T17:48:21Z)
• UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation [5.286124283204446]
  We propose UniCode$2$, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization.<n>By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity.<n>UniCode2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.
  arXiv  Detail & Related papers  (2025-06-25T07:57:09Z)
• Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization [20.063863466319326]
  SignViP is a novel framework that incorporates multiple fine-grained conditions for improved generation fidelity.<n>SignViP achieves state-of-the-art performance across metrics, including video quality temporal coherence, and semantic fidelity.
  arXiv  Detail & Related papers  (2025-06-19T02:56:06Z)
• Perception Encoder: The best visual embeddings are not at the output of the network [70.86738083862099]
  We introduce Perception (PE), a vision encoder for image and video understanding trained via simple vision-language learning.<n>We find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks.<n>Together, our PE family of models achieves best-in-class results on a wide variety of tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:59:57Z)
• Unveiling Encoder-Free Vision-Language Models [62.52803514667452]
  Existing vision-language models (VLMs) mostly rely on vision encoders to extract visual features followed by large language models (LLMs) for visual-language tasks. We bridge the gap between encoder-based and encoder-free models, and present a simple yet effective training recipe towards pure VLMs. We launch EVE, an encoder-free vision-language model that can be trained and forwarded efficiently.
  arXiv  Detail & Related papers  (2024-06-17T17:59:44Z)
• Do Vision and Language Encoders Represent the World Similarly? [22.70701869402434]
  Aligned text-image encoders such as CLIP have become the de facto model for vision-language tasks. We find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists without any training.
  arXiv  Detail & Related papers  (2024-01-10T15:51:39Z)
• CogVLM: Visual Expert for Pretrained Language Models [56.69978233342978]
  We introduce CogVLM, a powerful open-source visual language foundation model. CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC.
  arXiv  Detail & Related papers  (2023-11-06T13:04:39Z)
• i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data [101.52821120195975]
  i-Code V2 is first model capable of generating natural language from any combination of Vision, Language, and Speech data. System is pretrained end-to-end on a large collection of dual- and single-modality datasets.
  arXiv  Detail & Related papers  (2023-05-21T01:25:44Z)
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation [65.6846553962117]
  We present UniXcoder, a unified cross-modal pre-trained model for programming language. We propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. We evaluate UniXcoder on five code-related tasks over nine datasets.
  arXiv  Detail & Related papers  (2022-03-08T04:48:07Z)
• Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network [99.03895740754402]
  We propose a two-stream decoupled design of encoder-decoder structure, in which two decoupled cross-modal encoder and decoder are involved. As an alternative, we propose a primary scheduled sampling strategy that mitigates such discrepancy via pretraining encoder-decoder in a two-pass manner.
  arXiv  Detail & Related papers  (2021-01-27T17:36:57Z)
• Fast Interleaved Bidirectional Sequence Generation [90.58793284654692]
  We introduce a decoder that generates target words from the left-to-right and right-to-left directions simultaneously. We show that we can easily convert a standard architecture for unidirectional decoding into a bidirectional decoder. Our interleaved bidirectional decoder (IBDecoder) retains the model simplicity and training efficiency of the standard Transformer.
  arXiv  Detail & Related papers  (2020-10-27T17:38:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.