Is the Modality Gap a Bug or a Feature? A Robustness Perspective

• URL: http://arxiv.org/abs/2603.29080v1
• Date: Mon, 30 Mar 2026 23:41:22 GMT
• Title: Is the Modality Gap a Bug or a Feature? A Robustness Perspective
• Abstract summary: Almost all modern multi-modal models seek an embedding space in which the two modalities are aligned.<n>We show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated.<n>We also show that under these conditions the modality gap is monotonically related to robustness.
• Score: 12.228713815185705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings [91.3041144166326]
  We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:39:40Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap [4.437949196235149]
  A modality gap has been reported in two-encoder contrastive models like CLIP. We show that even when accounting for all these factors, the contrastive loss actually creates a gap during training. We present evidence that attributes this contrastive gap to low uniformity in CLIP space, resulting in embeddings that occupy only a small portion of the latent space.
  arXiv  Detail & Related papers  (2024-05-28T20:28:07Z)
• Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models [82.8261101680427]
  Smooth latent spaces ensure that a perturbation on an input latent corresponds to a steady change in the output image. This property proves beneficial in downstream tasks, including image inversion, inversion, and editing. We propose Smooth Diffusion, a new category of diffusion models that can be simultaneously high-performing and smooth.
  arXiv  Detail & Related papers  (2023-12-07T16:26:23Z)
• Geodesic Multi-Modal Mixup for Robust Fine-Tuning [21.298732743643168]
  We show that CLIP retains poor uniformity and alignment even after fine-tuning. We propose a Geodesic Multi-Modal Mixup that mixes the embeddings of image and text to generate hard negative samples. Our method provides transferable representations, enabling robust model adaptation on diverse tasks.
  arXiv  Detail & Related papers  (2022-03-08T07:34:52Z)
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning [35.25854322376364]
  We show that different data modalities are embedded at arm's length in their shared representation in multi-modal models such as CLIP. In contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness.
  arXiv  Detail & Related papers  (2022-03-03T22:53:54Z)
• Exploring Data Augmentation for Multi-Modality 3D Object Detection [82.9988604088494]
  It is counter-intuitive that multi-modality methods based on point cloud and images perform only marginally better or sometimes worse than approaches that solely use point cloud. We propose a pipeline, named transformation flow, to bridge the gap between single and multi-modality data augmentation with transformation reversing and replaying. Our method also wins the best PKL award in the 3rd nuScenes detection challenge.
  arXiv  Detail & Related papers  (2020-12-23T15:23:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.