CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization

• URL: http://arxiv.org/abs/2503.24182v1
• Date: Mon, 31 Mar 2025 15:00:01 GMT
• Title: CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization
• Authors: Yingrui Ji, Xi Xiao, Gaofei Chen, Hao Xu, Chenrui Ma, Lijing Zhu, Aokun Liang, Jiansheng Chen,
• Abstract summary: We propose the Cross-modal Information Bottleneck (CIB) framework as an implicit Information Bottleneck optimization.<n>Under this view, the model maximizes shared cross-modal information while discarding modality-specific redundancies.<n>We introduce a Cross-modal Information Bottleneck Regularization (CIBR) method that explicitly enforces these IB principles during training.
• Score: 13.867420348797783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success in cross-modal tasks such as zero-shot image classification and text-image retrieval by effectively aligning visual and textual representations. However, the theoretical foundations underlying CLIP's strong generalization remain unclear. In this work, we address this gap by proposing the Cross-modal Information Bottleneck (CIB) framework. CIB offers a principled interpretation of CLIP's contrastive learning objective as an implicit Information Bottleneck optimization. Under this view, the model maximizes shared cross-modal information while discarding modality-specific redundancies, thereby preserving essential semantic alignment across modalities. Building on this insight, we introduce a Cross-modal Information Bottleneck Regularization (CIBR) method that explicitly enforces these IB principles during training. CIBR introduces a penalty term to discourage modality-specific redundancy, thereby enhancing semantic alignment between image and text features. We validate CIBR on extensive vision-language benchmarks, including zero-shot classification across seven diverse image datasets and text-image retrieval on MSCOCO and Flickr30K. The results show consistent performance gains over standard CLIP. These findings provide the first theoretical understanding of CLIP's generalization through the IB lens. They also demonstrate practical improvements, offering guidance for future cross-modal representation learning.

Related papers

• Data-Efficient Generalization for Zero-shot Composed Image Retrieval [67.46975191141928]
  ZS-CIR aims to retrieve the target image based on a reference image and a text description without requiring in-distribution triplets for training.<n>One prevalent approach follows the vision-language pretraining paradigm that employs a mapping network to transfer the image embedding to a pseudo-word token in the text embedding space.<n>We propose a Data-efficient Generalization (DeG) framework, including two novel designs, namely, Textual Supplement (TS) module and Semantic-Set (S-Set)
  arXiv  Detail & Related papers  (2025-03-07T07:49:31Z)
• Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability [15.155556606996994]
  Narrowing Information Bottleneck Theory is a novel framework that redefines the traditional bottleneck approach.<n>Our approach enhances image interpretability by an average of 9%, text interpretability by an average of 58.83%, and accelerates processing speed by 63.95%.
  arXiv  Detail & Related papers  (2025-02-16T19:01:37Z)
• Fully Aligned Network for Referring Image Segmentation [22.40918154209717]
  This paper focuses on the Referring Image task, which aims to segment objects from an image based on a given language description. The critical problem of RIS is achieving fine-grained alignment between different modalities to recognize and segment the target object. We present a Fully Aligned Network (FAN) that follows four cross-modal interaction principles.
  arXiv  Detail & Related papers  (2024-09-29T06:13:34Z)
• Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP [84.90129481336659]
  We study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets.
  arXiv  Detail & Related papers  (2023-10-02T06:41:30Z)
• Extending CLIP's Image-Text Alignment to Referring Image Segmentation [48.26552693472177]
  Referring Image (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. We propose RISCLIP, a novel framework that effectively leverages the cross-modal nature of CLIP for RIS.
  arXiv  Detail & Related papers  (2023-06-14T13:27:28Z)
• Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment [23.072180427273544]
  We discuss that directly employing CLIP for zero-shot image captioning relies more on the textual modality in context and largely ignores the visual information. To address this, we propose Cross-modal Language Models (CLMs) to facilitate unsupervised cross-modal learning. Experiments on MS COCO and Flickr 30K validate the promising performance of proposed approach in both captioning quality and computational efficiency.
  arXiv  Detail & Related papers  (2022-11-14T11:12:19Z)
• CLIP-Driven Fine-grained Text-Image Person Re-identification [50.94827165464813]
  TIReID aims to retrieve the image corresponding to the given text query from a pool of candidate images. We propose a CLIP-driven Fine-grained information excavation framework (CFine) to fully utilize the powerful knowledge of CLIP for TIReID.
  arXiv  Detail & Related papers  (2022-10-19T03:43:12Z)
• Non-Contrastive Learning Meets Language-Image Pre-Training [145.6671909437841]
  We study the validity of non-contrastive language-image pre-training (nCLIP) We introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics.
  arXiv  Detail & Related papers  (2022-10-17T17:57:46Z)
• Vision-Language Pre-Training with Triple Contrastive Learning [45.80365827890119]
  We propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Ours is the first work that takes into account local structure information for multi-modality representation learning.
  arXiv  Detail & Related papers  (2022-02-21T17:54:57Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.