Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

• URL: http://arxiv.org/abs/2502.04263v1
• Date: Thu, 06 Feb 2025 17:58:59 GMT
• Title: Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
• Authors: Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, Andrew D. Bagdanov,
• Abstract summary: Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance.
• Score: 13.696706205837238
• License:
• Abstract: Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like image-to-image retrieval. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra-modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.

Related papers

• MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching [54.740256498985026]
  Keypoint detection and description methods often struggle with multimodal data. We propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching.
  arXiv  Detail & Related papers  (2025-01-20T06:56:30Z)
• MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training [62.843316348659165]
  Deep learning-based image matching algorithms have dramatically outperformed humans in rapidly and accurately finding large amounts of correspondences. We propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals to train models to recognize and match fundamental structures across images. Our key finding is that the matching model trained with our framework achieves remarkable generalizability across more than eight unseen cross-modality registration tasks.
  arXiv  Detail & Related papers  (2025-01-13T18:37:36Z)
• From Unimodal to Multimodal: Scaling up Projectors to Align Modalities [16.733970553781887]
  We propose a novel approach that aligns vision and language modalities using only projection layers on pretrained, frozen unimodal encoders. Our method exploits the high semantic similarity between embedding spaces of well-trained vision and language models. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple projectors.
  arXiv  Detail & Related papers  (2024-09-28T17:57:32Z)
• Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment [2.389598109913754]
  We focus on Contrastive Language-Image Pre-training (CLIP), an open-vocabulary foundation model, which achieves high accuracy across many image classification tasks. There are still domains where zero-shot CLIP performance is far from optimal, such as Remote Sensing (RS) and medical imagery. We propose a methodology for the purpose of aligning distinct RS imagery modalities with the visual and textual modalities of CLIP.
  arXiv  Detail & Related papers  (2024-02-15T09:31:07Z)
• Multi-Modal Representation Learning with Text-Driven Soft Masks [48.19806080407593]
  We propose a visual-linguistic representation learning approach within a self-supervised learning framework. We generate diverse features for the image-text matching (ITM) task via soft-masking the regions in an image. We identify the relevant regions to each word by computing the word-conditional visual attention using multi-modal encoder.
  arXiv  Detail & Related papers  (2023-04-03T05:07:49Z)
• Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts [63.84720380390935]
  There exist two typical types, textiti.e., the fusion-encoder type and the dual-encoder type, depending on whether a heavy fusion module is used. We propose an effective yet straightforward scheme named PTUnifier to unify the two types. We first unify the input format by introducing visual and textual prompts, which serve as a feature bank that stores the most representative images/texts.
  arXiv  Detail & Related papers  (2023-02-17T15:43:42Z)
• Improving Cross-modal Alignment for Text-Guided Image Inpainting [36.1319565907582]
  Text-guided image inpainting (TGII) aims to restore missing regions based on a given text in a damaged image. We propose a novel model for TGII by improving cross-modal alignment. Our model achieves state-of-the-art performance compared with other strong competitors.
  arXiv  Detail & Related papers  (2023-01-26T19:18:27Z)
• CLIP-Driven Fine-grained Text-Image Person Re-identification [50.94827165464813]
  TIReID aims to retrieve the image corresponding to the given text query from a pool of candidate images. We propose a CLIP-driven Fine-grained information excavation framework (CFine) to fully utilize the powerful knowledge of CLIP for TIReID.
  arXiv  Detail & Related papers  (2022-10-19T03:43:12Z)
• ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training [40.05046655477684]
  ERNIE-ViL 2.0 is a Multi-View Contrastive learning framework to build intra-modal and inter-modal correlations between diverse views simultaneously. We construct sequences of object tags as a special textual view to narrow the cross-modal semantic gap on noisy image-text pairs. ERNIE-ViL 2.0 achieves competitive results on English cross-modal retrieval.
  arXiv  Detail & Related papers  (2022-09-30T07:20:07Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.