CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally

• URL: http://arxiv.org/abs/2502.03566v2
• Date: Sat, 08 Feb 2025 14:04:11 GMT
• Title: CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
• Abstract summary: Recent studies have questioned CLIP's ability to represent compositional concepts effectively.<n>We propose Linear Attribute Binding CLIP or LABCLIP to improve CLIP's compositional understanding.
• Score: 16.067527644904434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks. However, recent studies have questioned its ability to represent compositional concepts effectively. These works suggest that CLIP often acts like a bag-of-words (BoW) model, interpreting images and text as sets of individual concepts without grasping the structural relationships. In particular, CLIP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text. In this work, we investigate why CLIP exhibits this BoW-like behavior. We find that the correct attribute-object binding information is already present in individual text and image modalities. Instead, the issue lies in the cross-modal alignment, which relies on cosine similarity. To address this, we propose Linear Attribute Binding CLIP or LABCLIP. It applies a linear transformation to text embeddings before computing cosine similarity. This approach significantly improves CLIP's ability to bind attributes to correct objects, thereby enhancing its compositional understanding. The code is available at https://github.com/kdariina/CLIP-not-BoW-unimodally.

Related papers

• ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching [9.610261779024219]
  ABE-CLIP is a training-free Attribute Binding Enhancement method designed to strengthen attribute-object binding in CLIP-like models.<n>We employ a Semantic Refinement Mechanism to refine token embeddings for both object and attribute phrases in the text.<n>By aggregating localized similarity scores, ABE-CLIP computes the final image-text similarity.
  arXiv  Detail & Related papers  (2025-12-19T02:36:51Z)
• CLIP Won't Learn Object-Attribute Binding from Natural Data and Here is Why [28.410388568106587]
  We show that low attribute density, incomplete captions, and the saliency bias have a detrimental effect on binding performance.<n>We find that neither scaling the batch size, i.e., implicitly adding more hard negatives, nor explicitly creating hard negatives enables CLIP to learn reliable binding.
  arXiv  Detail & Related papers  (2025-07-10T17:57:31Z)
• Is CLIP ideal? No. Can we fix it? Yes! [30.71718499767702]
  Contrastive Language-Image Pre-Training is a popular method for learning multimodal latent spaces with well-organized semantics. Despite its wide range of applications, CLIP's latent space is known to fail at handling complex visual-textual interactions. We propose Cosine Similarity Maps (DCSMs) as a principled and interpretable scoring method for CLIP-like models.
  arXiv  Detail & Related papers  (2025-03-10T23:42:04Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Finetuning CLIP to Reason about Pairwise Differences [52.028073305958074]
  We propose an approach to train vision-language models such as CLIP in a contrastive manner to reason about differences in embedding space. We first demonstrate that our approach yields significantly improved capabilities in ranking images by a certain attribute. We also illustrate that the resulting embeddings obey a larger degree of geometric properties in embedding space.
  arXiv  Detail & Related papers  (2024-09-15T13:02:14Z)
• Mining Open Semantics from CLIP: A Relation Transition Perspective for Few-Shot Learning [46.25534556546322]
  We propose to mine open semantics as anchors to perform a relation transition from image-anchor relationship to image-target relationship to make predictions. Our method performs favorably against previous state-of-the-arts considering few-shot classification settings.
  arXiv  Detail & Related papers  (2024-06-17T06:28:58Z)
• Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
  We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. Experiment results across various datasets and models confirm CODER's effectiveness.
  arXiv  Detail & Related papers  (2024-04-27T02:04:36Z)
• PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts [33.109305627550405]
  This paper draws inspiration from the human visual perception process. We propose a training-free, two-step zero-shot classification method PerceptionCLIP. Our experiments show that PerceptionCLIP achieves better generalization, group robustness, and interoperability.
  arXiv  Detail & Related papers  (2023-08-02T17:57:25Z)
• From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping [20.67415815472257]
  We propose a zero-shot method from association to generation for image captioning and video captioning. Knight State-of-the-Art achieves performance in zero-shot methods for image captioning and video captioning.
  arXiv  Detail & Related papers  (2023-04-26T04:06:20Z)
• Does CLIP Bind Concepts? Probing Compositionality in Large Image Models [24.85238978067213]
  We focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts. We benchmark them on three synthetic datasets designed to test concept binding. We find that CLIP can compose concepts in a single-object setting, but in situations where concept binding is needed, performance drops dramatically.
  arXiv  Detail & Related papers  (2022-12-20T18:46:28Z)
• Fine-tuned CLIP Models are Efficient Video Learners [54.96069171726668]
  Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos.
  arXiv  Detail & Related papers  (2022-12-06T18:59:58Z)
• CLIP2GAN: Towards Bridging Text with the Latent Space of GANs [128.47600914674985]
  We propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN.
  arXiv  Detail & Related papers  (2022-11-28T04:07:17Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)
• Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching [102.62343739435289]
  Existing image-text matching approaches infer the similarity of an image-text pair by capturing and aggregating the affinities between the text and each independent object of the image. We propose a Dual Path Recurrent Neural Network (DP-RNN) which processes images and sentences symmetrically by recurrent neural networks (RNN) Our model achieves the state-of-the-art performance on Flickr30K dataset and competitive performance on MS-COCO dataset.
  arXiv  Detail & Related papers  (2020-02-20T00:51:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.