Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training

• URL: http://arxiv.org/abs/2410.07336v1
• Date: Wed, 9 Oct 2024 18:00:09 GMT
• Title: Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training
• Authors: Sara Sarto, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara,
• Abstract summary: PAC-S++ is a learnable metric that leverages the CLIP model, pre-trained on both web-collected and cleaned data. We show that integrating PAC-S++ into the fine-tuning stage of a captioning model results in semantically richer captions with fewer repetitions and grammatical errors.
• Score: 44.008094698200026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite significant advancements in caption generation, existing evaluation metrics often fail to capture the full quality or fine-grained details of captions. This is mainly due to their reliance on non-specific human-written references or noisy pre-training data. Still, finding an effective metric is crucial not only for captions evaluation but also for the generation phase. Metrics can indeed play a key role in the fine-tuning stage of captioning models, ultimately enhancing the quality of the generated captions. In this paper, we propose PAC-S++, a learnable metric that leverages the CLIP model, pre-trained on both web-collected and cleaned data and regularized through additional pairs of generated visual and textual positive samples. Exploiting this stronger and curated pre-training, we also apply PAC-S++ as a reward in the Self-Critical Sequence Training (SCST) stage typically employed to fine-tune captioning models. Extensive experiments on different image and video datasets highlight the effectiveness of PAC-S++ compared to popular metrics for the task, including its sensitivity to object hallucinations. Furthermore, we show that integrating PAC-S++ into the fine-tuning stage of a captioning model results in semantically richer captions with fewer repetitions and grammatical errors. Evaluations on out-of-domain benchmarks further demonstrate the efficacy of our fine-tuning approach in enhancing model capabilities. Source code and trained models are publicly available at: https://github.com/aimagelab/pacscore.

Related papers

• Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization [44.008094698200026]
  We propose a new training paradigm termed Direct CLIP-Based Optimization (DiCO) Our approach jointly learns and optimize a reward model that is distilled from a learnable captioning evaluator with high human correlation. DiCO not only exhibits improved stability and enhanced quality in the generated captions but also aligns more closely with human preferences compared to existing methods.
  arXiv  Detail & Related papers  (2024-08-26T18:00:33Z)
• ALIP: Adaptive Language-Image Pre-training with Synthetic Caption [78.93535202851278]
  Contrastive Language-Image Pre-training (CLIP) has significantly boosted the performance of various vision-language tasks. The presence of intrinsic noise and unmatched image-text pairs in web data can potentially affect the performance of representation learning. We propose an Adaptive Language-Image Pre-training (ALIP), a bi-path model that integrates supervision from both raw text and synthetic caption.
  arXiv  Detail & Related papers  (2023-08-16T15:19:52Z)
• Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion [17.99150939602917]
  State-of-The-Art (SoTA) image captioning models often rely on the Microsoft COCO (MS-COCO) dataset for training. We present a novel approach to address previous challenges by showcasing how captions generated from different SoTA models can be effectively fused.
  arXiv  Detail & Related papers  (2023-06-20T15:13:02Z)
• Paraphrasing Is All You Need for Novel Object Captioning [126.66301869607656]
  Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. We present Paraphrasing-to-Captioning (P2C), a two-stage learning framework for NOC, which wouldally optimize the output captions via paraphrasing.
  arXiv  Detail & Related papers  (2022-09-25T22:56:04Z)
• Prompt-based Learning for Unpaired Image Captioning [86.44188293709307]
  Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Recent successes of Vision-Language Pre-Trained Models (VL-PTMs) have triggered the development of prompt-based learning. We present in this paper a novel scheme based on prompt to train the UIC model, making best use of the powerful generalization ability.
  arXiv  Detail & Related papers  (2022-05-26T03:13:43Z)
• Fine-grained Image Captioning with CLIP Reward [104.71533106301598]
  We propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text encoder to improve grammar that does not require extra text annotation. In experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model.
  arXiv  Detail & Related papers  (2022-05-26T02:46:09Z)
• CLIP Meets Video Captioners: Attribute-Aware Representation Learning Promotes Accurate Captioning [34.46948978082648]
  ImageNet Pre-training (INP) is usually used to help encode the video content, and a task-oriented network is fine-tuned from scratch to cope with caption generation. This paper investigates the potential deficiencies of INP for video captioning and explores the key to generating accurate descriptions. We introduce Dual Attribute Prediction, an auxiliary task requiring a video caption model to learn the correspondence between video content and attributes.
  arXiv  Detail & Related papers  (2021-11-30T06:37:44Z)
• Better Captioning with Sequence-Level Exploration [60.57850194028581]
  We show the limitation of the current sequence-level learning objective for captioning tasks. In theory, we show that the current objective is equivalent to only optimizing the precision side of the caption set. Empirical result shows that the model trained by this objective tends to get lower score on the recall side.
  arXiv  Detail & Related papers  (2020-03-08T09:08:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.