ALIP: Adaptive Language-Image Pre-training with Synthetic Caption

• URL: http://arxiv.org/abs/2308.08428v2
• Date: Fri, 18 Aug 2023 04:04:30 GMT
• Title: ALIP: Adaptive Language-Image Pre-training with Synthetic Caption
• Authors: Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, Tongliang Liu
• Abstract summary: Contrastive Language-Image Pre-training (CLIP) has significantly boosted the performance of various vision-language tasks. The presence of intrinsic noise and unmatched image-text pairs in web data can potentially affect the performance of representation learning. We propose an Adaptive Language-Image Pre-training (ALIP), a bi-path model that integrates supervision from both raw text and synthetic caption.
• Score: 78.93535202851278
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive Language-Image Pre-training (CLIP) has significantly boosted the performance of various vision-language tasks by scaling up the dataset with image-text pairs collected from the web. However, the presence of intrinsic noise and unmatched image-text pairs in web data can potentially affect the performance of representation learning. To address this issue, we first utilize the OFA model to generate synthetic captions that focus on the image content. The generated captions contain complementary information that is beneficial for pre-training. Then, we propose an Adaptive Language-Image Pre-training (ALIP), a bi-path model that integrates supervision from both raw text and synthetic caption. As the core components of ALIP, the Language Consistency Gate (LCG) and Description Consistency Gate (DCG) dynamically adjust the weights of samples and image-text/caption pairs during the training process. Meanwhile, the adaptive contrastive loss can effectively reduce the impact of noise data and enhances the efficiency of pre-training data. We validate ALIP with experiments on different scales of models and pre-training datasets. Experiments results show that ALIP achieves state-of-the-art performance on multiple downstream tasks including zero-shot image-text retrieval and linear probe. To facilitate future research, the code and pre-trained models are released at https://github.com/deepglint/ALIP.

Related papers

• NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training [6.34265125858783]
  We propose a noise-robust framework for efficient vision-language pre-training that requires less pre-training data. Specifically, we bridge the modality gap between a frozen image encoder and a large language model with a transformer. We introduce two innovative learning strategies: noise-adaptive learning and concept-enhanced learning.
  arXiv  Detail & Related papers  (2024-09-15T01:54:17Z)
• Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning [78.19528555505961]
  We propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets, but can also leverage interleaved pre-training data.
  arXiv  Detail & Related papers  (2024-06-11T17:59:35Z)
• CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data [40.88256210436378]
  We present a novel weakly supervised pre-training of vision models on web-scale image-text data. The proposed method reframes pre-training on image-text data as a classification task. It achieves a remarkable $2.7times$ acceleration in training speed compared to contrastive learning on web-scale data.
  arXiv  Detail & Related papers  (2024-04-24T05:13:28Z)
• ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations [43.323791505213634]
  ASPIRE (Language-guided Data Augmentation for SPurIous correlation REmoval) is a solution for supplementing the training dataset with images without spurious features. It can generate non-spurious images without requiring any group labeling or existing non-spurious images in the training set. It improves the worst-group classification accuracy of prior methods by 1% - 38%.
  arXiv  Detail & Related papers  (2023-08-19T20:18:15Z)
• COSA: Concatenated Sample Pretrained Vision-Language Foundation Model [78.32081709802873]
  Most vision-language foundation models employ image-text datasets for pretraining. We propose COSA, a COncatenated SAmple pretrained vision-language foundation model. We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effectively converts existing image-text corpora into a pseudo long-form video-paragraph corpus.
  arXiv  Detail & Related papers  (2023-06-15T12:29:42Z)
• NLIP: Noise-robust Language-Image Pre-training [95.13287735264937]
  We propose a principled Noise-robust Language-Image Pre-training framework (NLIP) to stabilize pre-training via two schemes: noise-harmonization and noise-completion. Our NLIP can alleviate the common noise effects during image-text pre-training in a more efficient way.
  arXiv  Detail & Related papers  (2022-12-14T08:19:30Z)
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation [86.4572981982407]
  We propose BLIP, a new vision-language framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.
  arXiv  Detail & Related papers  (2022-01-28T12:49:48Z)
• VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning [128.6138588412508]
  This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. Our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects.
  arXiv  Detail & Related papers  (2020-09-28T23:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.