Accessing Vision Foundation Models via ImageNet-1K

• URL: http://arxiv.org/abs/2407.10366v2
• Date: Tue, 11 Feb 2025 18:44:46 GMT
• Title: Accessing Vision Foundation Models via ImageNet-1K
• Authors: Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, Yun Fu,
• Abstract summary: Proteus is trained at ImageNet-level costs with surprising ability, facilitating the accessibility of training foundation models for the broader research community. Proteus-L/14 matches the performance of the Oracle method DINOv2-L/14 across 19 benchmarks and outperforms other vision foundation models including CLIP-L/14 (400M), OpenCLIP-L/14 (400M/2B) and SynCLR-L/14 (600M) with a significantly smaller training set of 1.2M images.
• Score: 51.521125501182816
• License:
• Abstract: Vision foundation models are renowned for the generalization ability due to massive training data. Nevertheless, they demand tremendous training resources, and the training data is often inaccessible, e.g., CLIP, DINOv2, posing great challenges to developing derivatives that could facilitate the research. In this work, we offer a very simple and general solution, named \textit{Proteus}, to distill foundation models into smaller equivalents on ImageNet-1K without access to the original training data. Specifically, we remove the designs from conventional knowledge distillation settings that result in dataset bias and present three levels of training objectives, i.e., token, patch, and feature, to maximize the efficacy of knowledge transfer. In this manner, Proteus is trained at ImageNet-level costs with surprising ability, facilitating the accessibility of training foundation models for the broader research community. When leveraging DINOv2-g/14 as the teacher, Proteus-L/14 matches the performance of the Oracle method DINOv2-L/14 (142M training data) across 19 benchmarks and outperforms other vision foundation models including CLIP-L/14 (400M), OpenCLIP-L/14 (400M/2B) and SynCLR-L/14 (600M) with a significantly smaller training set of 1.2M images.

Related papers

• Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging [1.533133219129073]
  Data scarcity is a major limiting factor for applying modern machine learning techniques to clinical tasks. We conducted a benchmark study of few-shot learning and zero-shot learning using 16 pretrained foundation models on 19 diverse medical imaging datasets. Our results indicate that BiomedCLIP, a model pretrained exclusively on medical data, performs best on average for very small training set sizes.
  arXiv  Detail & Related papers  (2024-08-15T09:55:51Z)
• Effective pruning of web-scale datasets based on complexity of concept clusters [48.125618324485195]
  We present a method for pruning large-scale multimodal datasets for training CLIP-style models on ImageNet. We find that training on a smaller set of high-quality data can lead to higher performance with significantly lower training costs. We achieve a new state-of-the-art Imagehttps://info.arxiv.org/help/prep#commentsNet zero-shot accuracy and a competitive average zero-shot accuracy on 38 evaluation tasks.
  arXiv  Detail & Related papers  (2024-01-09T14:32:24Z)
• Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models [41.292216950622084]
  Vision Foundation Models (VFMs) pretrained on massive datasets exhibit impressive performance on various downstream tasks. Due to their high inference compute cost, these models cannot be deployed for many real-world applications. We propose a simple task-oriented knowledge transfer approach as a highly effective solution to this problem.
  arXiv  Detail & Related papers  (2023-11-30T04:07:44Z)
• MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training [17.158498267947877]
  We introduce MobileCLIP, a new family of efficient image-text models optimized for runtime performance. MobileCLIP uses knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset.
  arXiv  Detail & Related papers  (2023-11-28T18:55:42Z)
• UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks [60.46473247205654]
  Using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models.
  arXiv  Detail & Related papers  (2023-06-07T18:26:22Z)
• DIME-FM: DIstilling Multimodal and Efficient Foundation Models [72.1900621000677]
  Large Vision-Language Foundation Models (VLFM) are trained on large-scale datasets of image-caption pairs. We introduce a new distillation mechanism (DIME-FM) that allows us to transfer the knowledge contained in large VLFMs to smaller, customized foundation models. The resulting model "Distill-ViT-B/32" rivals the CLIP-ViT-B/32 model pre-trained on its private WiT dataset.
  arXiv  Detail & Related papers  (2023-03-31T17:47:23Z)
• EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones [80.662250618795]
  This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers) As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models by >1.5x on ImageNet-1K/22K without sacrificing accuracy.
  arXiv  Detail & Related papers  (2022-11-17T17:38:55Z)
• A Deeper Look at Salient Object Detection: Bi-stream Network with a Small Training Dataset [62.26677215668959]
  We provide a feasible way to construct a novel small-scale training set, which only contains 4K images. We propose a novel bi-stream network to take full advantage of our proposed small training set.
  arXiv  Detail & Related papers  (2020-08-07T01:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.