L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training

• URL: http://arxiv.org/abs/2507.08710v1
• Date: Fri, 11 Jul 2025 16:08:12 GMT
• Title: L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training
• Abstract summary: We propose a novel embedding-based captioning metric computation termed as L-CLIPScore.<n>L-CLIPScore is calculated from a lightweight CLIP, which is a dual-encoder architecture compressed and distilled from CLIP.<n>We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality.
• Score: 40.87934127615281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.

Related papers

• Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance [23.886374573928965]
  We show that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades.<n>Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters.
  arXiv  Detail & Related papers  (2026-09-04T21:29:20Z)
• ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder [50.25233123718465]
  The CLIP text encoder is limited by a maximum input length of 77 tokens.<n>ProCLIP is a curriculum learning-based progressive vision-language alignment framework.
  arXiv  Detail & Related papers  (2025-10-21T16:48:49Z)
• un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP [75.19266107565109]
  Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks.<n>This work focuses on improving existing CLIP models, aiming to capture as many visual details in images as possible.
  arXiv  Detail & Related papers  (2025-05-30T12:29:38Z)
• Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation [4.063715077687089]
  Distill CLIP (DCLIP) is a fine-tuned variant of the CLIP model.<n>It enhances multimodal image-text retrieval while preserving the original model's strong zero-shot classification capabilities.
  arXiv  Detail & Related papers  (2025-05-25T07:08:07Z)
• DiffCLIP: Differential Attention Meets CLIP [57.396578974401734]
  We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures.<n>With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks.
  arXiv  Detail & Related papers  (2025-03-09T14:04:09Z)
• PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models [8.480318790780037]
  We propose a plug-in prompt-regularization method called PLPP, which use perplexity loss to regularize prompt learning.<n>Experiments conducted on four classification tasks indicate that PLPP exhibits superior performance compared to existing methods.
  arXiv  Detail & Related papers  (2024-12-18T03:08:53Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation [10.502680141980642]
  Open-vocabulary semantic segmentation seeks to label each pixel in an image with arbitrary text descriptions.<n> vision-language foundation models, especially CLIP, have emerged as powerful tools for acquiring open-vocabulary capabilities.<n>H-CLIP achieves new SOTA open-vocabulary semantic segmentation results while only requiring updating approximately 4% of the total parameters of CLIP.
  arXiv  Detail & Related papers  (2024-05-29T07:41:34Z)
• Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition [77.93678598476149]
  We establish a novel Symmetrical Linguistic Feature Distillation framework (named CLIP-OCR) By cascading the CLIP image encoder with the reversed CLIP text encoder, a symmetrical structure is built with an image-to-text feature flow. Extensive experiments demonstrate the effectiveness of CLIP-OCR with 93.8% average accuracy on six popular STR benchmarks.
  arXiv  Detail & Related papers  (2023-10-08T04:00:20Z)
• Linear Alignment of Vision-language Models for Image Captioning [8.921774238325566]
  We propose a lightweight captioning method called ReCap, which can be trained up to 1000 times faster than existing lightweight methods.<n>We also propose two new learning-based image-captioning metrics built on CLIP score along with our proposed alignment.
  arXiv  Detail & Related papers  (2023-07-10T17:59:21Z)
• ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation [35.60888272729273]
  Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a two-stage scheme. While effective, such a scheme requires two image encoders, one for proposal generation and one for CLIP, leading to a complicated pipeline and high computational cost. We propose a simpler-and-efficient one-stage solution that directly extends CLIP's zero-shot prediction capability from image to pixel level.
  arXiv  Detail & Related papers  (2022-12-07T12:05:00Z)
• Boosting Continuous Sign Language Recognition via Cross Modality Augmentation [135.30357113518127]
  Continuous sign language recognition deals with unaligned video-text pair. We propose a novel architecture with cross modality augmentation. The proposed framework can be easily extended to other existing CTC based continuous SLR architectures.
  arXiv  Detail & Related papers  (2020-10-11T15:07:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.