Centered Masking for Language-Image Pre-Training

• URL: http://arxiv.org/abs/2403.15837v2
• Date: Wed, 27 Mar 2024 08:54:06 GMT
• Title: Centered Masking for Language-Image Pre-Training
• Authors: Mingliang Liang, Martha Larson,
• Abstract summary: We introduce Gaussian masking for Language-Image Pre-Training (GLIP) GLIP is a novel, straightforward, and effective technique for masking image patches during pre-training of a vision-language model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We introduce Gaussian masking for Language-Image Pre-Training (GLIP) a novel, straightforward, and effective technique for masking image patches during pre-training of a vision-language model. GLIP builds on Fast Language-Image Pre-Training (FLIP), which randomly masks image patches while training a CLIP model. GLIP replaces random masking with centered masking, that uses a Gaussian distribution and is inspired by the importance of image patches at the center of the image. GLIP retains the same computational savings as FLIP, while improving performance across a range of downstream datasets and tasks, as demonstrated by our experimental results. We show the benefits of GLIP to be easy to obtain, requiring no delicate tuning of the Gaussian, and also applicable to data sets containing images without an obvious center focus.

Related papers

• Efficient Vision-Language Pre-training by Cluster Masking [13.845233914223561]
  We propose a simple strategy for masking image patches during visual-language contrastive learning. We randomly mask clusters of visually similar image patches, as measured by their raw pixel intensities. This provides an extra learning signal, beyond the contrastive training itself, since it forces a model to predict words for masked visual structures solely from context.
  arXiv  Detail & Related papers  (2024-05-14T17:59:40Z)
• ALIP: Adaptive Language-Image Pre-training with Synthetic Caption [78.93535202851278]
  Contrastive Language-Image Pre-training (CLIP) has significantly boosted the performance of various vision-language tasks. The presence of intrinsic noise and unmatched image-text pairs in web data can potentially affect the performance of representation learning. We propose an Adaptive Language-Image Pre-training (ALIP), a bi-path model that integrates supervision from both raw text and synthetic caption.
  arXiv  Detail & Related papers  (2023-08-16T15:19:52Z)
• Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training [59.923672191632065]
  We propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT) MaPeT employs autoregressive and permuted predictions to capture intra-patch dependencies. Our results demonstrate that MaPeT achieves competitive performance on ImageNet.
  arXiv  Detail & Related papers  (2023-06-12T18:12:19Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• Scaling Language-Image Pre-training via Masking [63.36988191660858]
  Fast Language-Image Pre-training (FLIP) is a simple and more efficient method for training CLIP. Masking allows us to learn from more image-text pairs given the same wall-clock time. FLIP dominantly outperforms CLIP counterparts trained on the same data.
  arXiv  Detail & Related papers  (2022-12-01T18:59:57Z)
• Leveraging per Image-Token Consistency for Vision-Language Pre-training [52.825150269820696]
  Cross-modal masked language modeling (CMLM) is insufficient for vision-language pre-training. We propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training) The proposed EPIC method is easily combined with pre-training methods.
  arXiv  Detail & Related papers  (2022-11-20T12:10:53Z)
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP [45.81698881151867]
  Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-
  arXiv  Detail & Related papers  (2022-10-09T02:57:32Z)
• MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining [138.86293836634323]
  MaskCLIP incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. MaskCLIP achieves superior results in linear probing, finetuning, and zero-shot performance with the guidance of the language encoder.
  arXiv  Detail & Related papers  (2022-08-25T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.