SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP

• URL: http://arxiv.org/abs/2509.26036v2
• Date: Wed, 01 Oct 2025 09:18:57 GMT
• Title: SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP
• Abstract summary: We introduce SeMoBridge, a lightweight yet powerful approach that directly addresses the misalignment.<n>Our method maps images into the text modality, while keeping their semantic content intact through what we call a Semantic Modality Bridge.<n> Experiments show that the trained version, SeMoBridge-T, requires only a fraction of the training time while overall outperforming other methods.
• Score: 6.427791334895624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Contrastive Language-Image Pretraining (CLIP) excels at zero-shot tasks by aligning image and text embeddings, its performance in few-shot classification is hindered by a critical limitation: intra-modal misalignment. This issue, caused by a persistent modality gap and CLIP's exclusively inter-modal training objective, leaves the embedding spaces uncalibrated, making direct image-to-image comparisons unreliable. Existing methods attempt to address this by refining similarity logits or by computationally expensive per-sample optimization. To overcome these challenges, we introduce SeMoBridge, a lightweight yet powerful approach that directly addresses the misalignment. Our method maps images into the text modality, while keeping their semantic content intact through what we call a Semantic Modality Bridge. SeMoBridge is closed-form and can optionally be trained through multi-modal supervision, combining image and text-alignment losses to optimize the projection. Experiments show that the trained version, SeMoBridge-T, requires only a fraction of the training time while overall outperforming other methods, particularly in low-data scenarios (1, 2, and 4 shots). The code is available at https://github.com/christti98/semobridge.

Related papers

• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding [56.38477453373368]
  LazyMCoT is a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty.<n>LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.
  arXiv  Detail & Related papers  (2026-06-15T03:17:44Z)
• Post-pre-training for Modality Alignment in Vision-Language Foundation Models [12.110530026601968]
  This paper presents CLIP-Refine, a post-pre-training method for CLIP models at a phase between pre-training and fine-tuning.<n>It aims to align the feature space with 1 epoch training on small image-text datasets without zero-shot performance degradations.
  arXiv  Detail & Related papers  (2025-04-17T07:46:19Z)
• Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection [54.21851618853518]
  We present a concise yet effective approach called Patch Generation-to-Selection to enhance CLIP's training efficiency.<n>Our approach, CLIP-PGS, sets new state-of-the-art results in zero-shot classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-21T12:10:38Z)
• I0T: Embedding Standardization Method Towards Zero Modality Gap [18.032635790176464]
  Contrastive Language-Image Pretraining (CLIP) enables zero-shot inference in downstream tasks such as image-text retrieval and classification.<n>Recent works extending CLIP suffer from the issue of modality gap.<n>We propose two methods to address the modality gap.
  arXiv  Detail & Related papers  (2024-12-18T22:35:01Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning [53.766434746801366]
  Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. Hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. We propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples.
  arXiv  Detail & Related papers  (2024-07-23T09:00:52Z)
• Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
  We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. Experiment results across various datasets and models confirm CODER's effectiveness.
  arXiv  Detail & Related papers  (2024-04-27T02:04:36Z)
• MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments [72.6405488990753]
  Self-supervised learning can be used for mitigating the greedy needs of Vision Transformer networks. We propose a single-stage and standalone method, MOCA, which unifies both desired properties. We achieve new state-of-the-art results on low-shot settings and strong experimental results in various evaluation protocols.
  arXiv  Detail & Related papers  (2023-07-18T15:46:20Z)
• Black Box Few-Shot Adaptation for Vision-Language models [41.49584259596654]
  Vision-Language (V-L) models trained with contrastive learning to align the visual and language modalities have been shown to be strong few-shot learners. We describe a black-box method for V-L few-shot adaptation that operates on pre-computed image and text features. We propose Linear Feature Alignment (LFA), a simple linear approach for V-L re-alignment in the target domain.
  arXiv  Detail & Related papers  (2023-04-04T12:42:29Z)
• Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning [88.5382122413913]
  We study whether language supervision can result in vision models with more transferable representations than traditional image-only methods. We find that image-only methods do not match CLIP's transfer performance, even when they are trained with more image data. Motivated by our findings, we devise simple prescriptions to enable CLIP to better leverage the language information present in existing pre-training datasets.
  arXiv  Detail & Related papers  (2022-07-15T17:50:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.