Learning with Unmasked Tokens Drives Stronger Vision Learners

• URL: http://arxiv.org/abs/2310.13593v3
• Date: Sat, 24 Aug 2024 08:46:20 GMT
• Title: Learning with Unmasked Tokens Drives Stronger Vision Learners
• Authors: Taekyung Kim, Sanghyuk Chun, Byeongho Heo, Dongyoon Han,
• Abstract summary: Masked image modeling (MIM) has become a leading self-supervised learning strategy. We improve MIM by explicitly incorporating unmasked tokens into the training process. We achieve 84.2% top-1 accuracy with ViT-B on ImageNet-1K with 0.6%p gain.
• Score: 39.752789949834536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked image modeling (MIM) has become a leading self-supervised learning strategy. MIMs such as Masked Autoencoder (MAE) learn strong representations by randomly masking input tokens for the encoder to process, with the decoder reconstructing the masked tokens to the input. However, MIM pre-trained encoders often exhibit a limited attention span, attributed to MIM's sole focus on regressing masked tokens only, which may impede the encoder's broader context learning. To tackle the limitation, we improve MIM by explicitly incorporating unmasked tokens into the training process. Specifically, our method enables the encoder to learn from broader context supervision, allowing unmasked tokens to experience broader contexts while the decoder reconstructs masked tokens. Thus, the encoded unmasked tokens are equipped with extensive contextual information, empowering masked tokens to leverage the enhanced unmasked tokens for MIM. As a result, our simple remedy trains more discriminative representations revealed by achieving 84.2% top-1 accuracy with ViT-B on ImageNet-1K with 0.6%p gain. We attribute the success to the enhanced pre-training method, as evidenced by the singular value spectrum and attention analyses. Finally, our models achieve significant performance gains at the downstream semantic segmentation and fine-grained visual classification tasks; and on diverse robust evaluation metrics. Code is available at https://github.com/naver-ai/lut

Related papers

• Enhancing Vision-Language Model with Unmasked Token Alignment [37.12838142681491]
  This paper introduces Unmasked Token Alignment (UTA), a method that leverages existing CLIP models to further enhance its vision-language representations. UTA trains a Vision Transformer (ViT) by aligning unmasked visual tokens to the corresponding image tokens from a frozen CLIP vision encoder, which automatically aligns the ViT model with the CLIP text encoder.
  arXiv  Detail & Related papers  (2024-05-29T11:48:17Z)
• SEP: Self-Enhanced Prompt Tuning for Visual-Language Model [93.94454894142413]
  We introduce a novel approach named Self-Enhanced Prompt Tuning (SEP) SEP explicitly incorporates discriminative prior knowledge to enhance both textual-level and visual-level embeddings. Comprehensive evaluations across various benchmarks and tasks confirm SEP's efficacy in prompt tuning.
  arXiv  Detail & Related papers  (2024-05-24T13:35:56Z)
• Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization [42.82742477950748]
  Masked Autoencoder (MAE) is a notable method for self-supervised pretraining in visual representation learning. We introduce the Multi-level Optimized Mask Autoencoder (MLO-MAE), a novel framework that learns an optimal masking strategy during pretraining. Our experimental findings highlight MLO-MAE's significant advancements in visual representation learning.
  arXiv  Detail & Related papers  (2024-02-28T07:37:26Z)
• Tokenize Anything via Prompting [65.93061853439512]
  We present a unified, promptable model capable of simultaneously segmenting, recognizing, and captioning anything. We train a generalizable model with massive segmentation masks, eg, SA-1B masks, and semantic priors from a pre-trained CLIP model with 5 billion parameters. We believe this model can be a versatile region-level image tokenizer, capable of encoding general-purpose region context.
  arXiv  Detail & Related papers  (2023-12-14T17:01:02Z)
• Leveraging per Image-Token Consistency for Vision-Language Pre-training [52.825150269820696]
  Cross-modal masked language modeling (CMLM) is insufficient for vision-language pre-training. We propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training) The proposed EPIC method is easily combined with pre-training methods.
  arXiv  Detail & Related papers  (2022-11-20T12:10:53Z)
• AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders [44.87786478095987]
  Masked Autoencoders learn general representations for image, text, audio, video, etc., by masked input data from tokens of the visible data. This paper proposes an adaptive masking strategy for MAEs that is end-to-end trainable. AdaMAE samples visible tokens based on the semantic context using an auxiliary sampling network.
  arXiv  Detail & Related papers  (2022-11-16T18:59:48Z)
• Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers [122.01591448013977]
  Masked image modeling (MIM) has demonstrated promising results on downstream tasks. In this paper, we investigate whether there exist other effective ways to learn by recovering missing contents' We summarize a few design principles for token-based pre-training of vision transformers. This design achieves superior performance over MIM in a series of downstream recognition tasks without extra computational cost.
  arXiv  Detail & Related papers  (2022-03-27T14:23:29Z)
• What to Hide from Your Students: Attention-Guided Masked Image Modeling [32.402567373491834]
  We argue that image token masking is fundamentally different from token masking in text. We introduce a novel masking strategy, called attention-guided masking (AttMask)
  arXiv  Detail & Related papers  (2022-03-23T20:52:50Z)
• Masked Autoencoders Are Scalable Vision Learners [60.97703494764904]
  Masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. Coupling these two designs enables us to train large models efficiently and effectively.
  arXiv  Detail & Related papers  (2021-11-11T18:46:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.