Classification Done Right for Vision-Language Pre-Training

• URL: http://arxiv.org/abs/2411.03313v2
• Date: Wed, 06 Nov 2024 12:07:08 GMT
• Title: Classification Done Right for Vision-Language Pre-Training
• Authors: Zilong Huang, Qinghao Ye, Bingyi Kang, Jiashi Feng, Haoqi Fan,
• Abstract summary: We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks.
• Score: 66.90286715149786
• License:
• Abstract: We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass

Related papers

• Finetuning CLIP to Reason about Pairwise Differences [52.028073305958074]
  We propose an approach to train vision-language models such as CLIP in a contrastive manner to reason about differences in embedding space. We first demonstrate that our approach yields significantly improved capabilities in ranking images by a certain attribute. We also illustrate that the resulting embeddings obey a larger degree of geometric properties in embedding space.
  arXiv  Detail & Related papers  (2024-09-15T13:02:14Z)
• Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery [50.564146730579424]
  We propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples. Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks.
  arXiv  Detail & Related papers  (2024-03-15T02:40:13Z)
• SILC: Improving Vision Language Pretraining with Self-Distillation [113.50400246862056]
  We introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation.
  arXiv  Detail & Related papers  (2023-10-20T08:44:47Z)
• LPN: Language-guided Prototypical Network for few-shot classification [16.37959398470535]
  Few-shot classification aims to adapt to new tasks with limited labeled examples. Recent methods explore suitable measures for the similarity between the query and support images. We propose a Language-guided Prototypical Network (LPN) for few-shot classification.
  arXiv  Detail & Related papers  (2023-07-04T06:54:01Z)
• DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training [73.74291217502928]
  We propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input.
  arXiv  Detail & Related papers  (2023-03-06T11:02:47Z)
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [91.56988987393483]
  We present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones.
  arXiv  Detail & Related papers  (2021-12-02T18:59:32Z)
• X-Class: Text Classification with Extremely Weak Supervision [39.25777650619999]
  In this paper, we explore text classification with extremely weak supervision. We propose a novel framework X-Class to realize the adaptive representations. X-Class can rival and even outperform seed-driven weakly supervised methods on 7 benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-24T06:09:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.