Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

• URL: http://arxiv.org/abs/2602.23588v1
• Date: Fri, 27 Feb 2026 01:38:58 GMT
• Title: Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning
• Abstract summary: HDFLIM is a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen.<n>We show that HDFLIM performance comparable to end-to-end vision-language training methods.
• Score: 0.8594140167290097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.

Related papers

• Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information [10.086588302174263]
  We propose a principled approach, termed deep modality-shared self-expressive model (DeepMORSE)<n>DeepMORSE discovers cross-modal structures via a modality-shared self-expressive model and simultaneously learns structured representations that conform to a union of modality-specific subspaces.<n>We evaluate our DeepMORSE on six widely used image clustering benchmarks and observe performance improvements exceeding 3%.
  arXiv  Detail & Related papers  (2026-08-09T02:23:49Z)
• Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness [16.541207762213272]
  Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations.<n>We introduce a novel finetuning framework that steers model reasoning toward concept-level semantics.<n>Our method improves robustness across multiple ViT-based models.
  arXiv  Detail & Related papers  (2026-03-09T12:31:14Z)
• Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models [84.78794648147608]
  A persistent geometric anomaly, the Modality Gap, remains.<n>Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions.<n>We propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap into stable biases and anisotropic residuals.<n>We then introduce ReAlign, a training-free modality alignment strategy.
  arXiv  Detail & Related papers  (2026-02-02T13:59:39Z)
• SITS-DECO: A Generative Decoder Is All You Need For Multitask Satellite Image Time Series Modelling [0.0]
  We introduce SITS-DECO, a proof-of-concept generative model that applies unified-sequence framing to EO data.<n>We show that the model can perform multiple supervised and self-supervised tasks within a single unified architecture.<n>Despite its simplicity and lack of spatial context, SITS-DECO outperforms much larger EO foundation models on crop-type classification.
  arXiv  Detail & Related papers  (2025-10-21T14:42:55Z)
• Global and Local Entailment Learning for Natural World Imagery [7.874291189886743]
  Radial Cross-Modal Embeddings (RCME) is a framework that enables the explicit modeling of transitivity-enforced entailment.<n>We develop a hierarchical vision-language foundation model capable of representing the hierarchy in the Tree of Life.
  arXiv  Detail & Related papers  (2025-06-26T17:05:06Z)
• Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution [88.20464308588889]
  We propose a Structural Similarity-Inspired Unfolding (SSIU) method for efficient image SR.<n>This method is designed through unfolding an SR optimization function constrained by structural similarity.<n>Our model outperforms current state-of-the-art models, boasting lower parameter counts and reduced memory consumption.
  arXiv  Detail & Related papers  (2025-06-13T14:29:40Z)
• Future Link Prediction Without Memory or Aggregation [25.066464612400768]
  Future link prediction on temporal graphs is a fundamental task with wide applicability in real-world dynamic systems.<n>Existing methods typically rely on complex memory and aggregation modules, yet struggle to handle unseen edges.<n>We propose Cross-Attention based Future Link Predictor on Temporal Graphs (CRAFT), a simple yet effective architecture that discards memory and aggregation modules.
  arXiv  Detail & Related papers  (2025-05-26T01:53:27Z)
• FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers [55.2480439325792]
  We propose FUSE, an approach to approximating an adapter layer that maps from one model's textual embedding space to another, even across different tokenizers. We show the efficacy of our approach via multi-objective optimization over vision-language and causal language models for image captioning and sentiment-based image captioning.
  arXiv  Detail & Related papers  (2024-08-09T02:16:37Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [49.80911683739506]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets.<n>We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation.<n>Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Switchable Representation Learning Framework with Self-compatibility [50.48336074436792]
  We propose a Switchable representation learning Framework with Self-Compatibility (SFSC) SFSC generates a series of compatible sub-models with different capacities through one training process. SFSC achieves state-of-the-art performance on the evaluated datasets.
  arXiv  Detail & Related papers  (2022-06-16T16:46:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.