Vision Backbone Efficient Selection for Image Classification in Low-Data Regimes

• URL: http://arxiv.org/abs/2410.08592v2
• Date: Tue, 19 Aug 2025 15:02:03 GMT
• Title: Vision Backbone Efficient Selection for Image Classification in Low-Data Regimes
• Abstract summary: We introduce dataset-specific backbone selection as a new research direction in computer vision.<n>We show that even simple search strategies can find well-suited backbones within a pool of over $1300$ pretrained models.<n>Our results show that even simple search strategies can find well-suited backbones within just ten minutes of search time on a single GPU.
• Score: 2.877753005139331
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transfer learning has become an essential tool in modern computer vision, allowing practitioners to leverage backbones, pretrained on large datasets, to train successful models from limited annotated data. Choosing the right backbone is crucial, especially for small datasets, since final performance depends heavily on the quality of the initial feature representations. While prior work has conducted benchmarks across various datasets to identify universal top-performing backbones, we demonstrate that backbone effectiveness is highly dataset-dependent, especially in low-data scenarios where no single backbone consistently excels. To overcome this limitation, we introduce dataset-specific backbone selection as a new research direction and investigate its practical viability in low-data regimes. Since exhaustive evaluation is computationally impractical for large backbone pools, we formalize Vision Backbone Efficient Selection (VIBES) as the problem of searching for high-performing backbones under computational constraints. We define the solution space, propose several heuristics, and demonstrate VIBES feasibility for low-data image classification by performing experiments on four diverse datasets. Our results show that even simple search strategies can find well-suited backbones within a pool of over $1300$ pretrained models, outperforming generic benchmark recommendations within just ten minutes of search time on a single GPU (NVIDIA RTX A5000).

Related papers

• Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies [50.39041754816285]
  We propose a framework for learning multi-validation weights that jointly adapts data selection to both the downstream task and the specific model.<n>Our approach achieves performance comparable to or exceeding full-dataset tuning while using only 30% of the training samples on GSM8K.
  arXiv  Detail & Related papers  (2026-05-10T17:30:16Z)
• Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning [70.57180215148125]
  Visual instruction tuning aims to enable large language models to comprehend the visual world.<n>Existing methods often grapple with the intractable trade-off between accuracy and efficiency.<n>We present LLaVA-Meteor, a novel approach that strategically compresses visual tokens without compromising core information.
  arXiv  Detail & Related papers  (2025-05-17T10:22:29Z)
• SkeletonX: Data-Efficient Skeleton-based Action Recognition via Cross-sample Feature Aggregation [34.65359766672547]
  This paper studies one-shot and limited-scale learning settings to enable efficient adaptation with minimal data.<n>We present SkeletonX, a lightweight training pipeline that integrates seamlessly with existing GCN-based skeleton action recognizers.<n>It surpasses previous state-of-the-art methods in the one-shot setting, with only 1/10 of the parameters and much fewer FLOPs.
  arXiv  Detail & Related papers  (2025-04-16T04:01:42Z)
• RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations [55.74675012171316]
  RELOCATE is a training-free baseline designed to perform the challenging task of visual query localization in long videos. To eliminate the need for task-specific training, RELOCATE leverages a region-based representation derived from pretrained vision models.
  arXiv  Detail & Related papers  (2024-12-02T18:59:53Z)
• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.<n>Data selection has shown promise in identifying the most representative samples from the entire dataset.<n>We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• Patch Ranking: Efficient CLIP by Learning to Rank Local Patches [11.225834286969283]
  Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP. We propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. We successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3 across seven datasets.
  arXiv  Detail & Related papers  (2024-09-22T22:04:26Z)
• AMAES: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentation [2.0749231618270803]
  This study investigates the impact of self-supervised pretraining of 3D semantic segmentation models on a large-scale, domain-specific dataset. We introduce BRAINS-45K, a dataset of 44,756 brain MRI volumes from public sources, the largest public dataset available.
  arXiv  Detail & Related papers  (2024-08-01T15:27:48Z)
• Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling [58.50618448027103]
  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. This paper explores the differences across various CLIP-trained vision backbones. Method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone.
  arXiv  Detail & Related papers  (2024-05-27T12:59:35Z)
• Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression [63.23578860867408]
  We investigate how to integrate the evaluations of importance and sparsity scores into a single stage. We present OFB, a cost-efficient approach that simultaneously evaluates both importance and sparsity scores. Experiments demonstrate that OFB can achieve superior compression performance over state-of-the-art searching-based and pruning-based methods.
  arXiv  Detail & Related papers  (2024-03-23T13:22:36Z)
• Less is More: High-value Data Selection for Visual Instruction Tuning [127.38740043393527]
  We propose a high-value data selection approach TIVE, to eliminate redundancy within the visual instruction data and reduce the training cost. Our approach using only about 15% data can achieve comparable average performance to the full-data fine-tuned model across eight benchmarks.
  arXiv  Detail & Related papers  (2024-03-14T16:47:25Z)
• Low-resource finetuning of foundation models beats state-of-the-art in histopathology [3.4577420145036375]
  We benchmark the most popular vision foundation models as feature extractors for histopathology data. By finetuning a foundation model on a single GPU for only two hours or three days depending on the dataset, we can match or outperform state-of-the-art feature extractors. This is a considerable shift from the current state, where only few institutions with large amounts of resources and datasets are able to train a feature extractor.
  arXiv  Detail & Related papers  (2024-01-09T18:46:59Z)
• Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances [49.631908848868505]
  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. We investigate the differences in CLIP performance among various neural architectures. We propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34%.
  arXiv  Detail & Related papers  (2023-12-22T03:01:41Z)
• Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks [139.3768582233067]
  Battle of the Backbones (BoB) is a benchmarking tool for neural network based computer vision systems. We find that vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular. In apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive.
  arXiv  Detail & Related papers  (2023-10-30T18:23:58Z)
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition [88.34182299496074]
  Action labels are only available on a source dataset, but unavailable on a target dataset in the training stage. We utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets. By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks.
  arXiv  Detail & Related papers  (2022-07-17T07:05:39Z)
• Compactness Score: A Fast Filter Method for Unsupervised Feature Selection [66.84571085643928]
  We propose a fast unsupervised feature selection method, named as, Compactness Score (CSUFS) to select desired features. Our proposed algorithm seems to be more accurate and efficient compared with existing algorithms.
  arXiv  Detail & Related papers  (2022-01-31T13:01:37Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• Unsupervised Pre-training for Person Re-identification [90.98552221699508]
  We present a large scale unlabeled person re-identification (Re-ID) dataset "LUPerson" We make the first attempt of performing unsupervised pre-training for improving the generalization ability of the learned person Re-ID feature representation.
  arXiv  Detail & Related papers  (2020-12-07T14:48:26Z)
• The Little W-Net That Could: State-of-the-Art Retinal Vessel Segmentation with Minimalistic Models [19.089445797922316]
  We show that a minimalistic version of a standard U-Net with several orders of magnitude less parameters closely approximates the performance of current best techniques. We also propose a simple extension, dubbed W-Net, which reaches outstanding performance on several popular datasets. We also test our approach on the Artery/Vein segmentation problem, where we again achieve results well-aligned with the state-of-the-art.
  arXiv  Detail & Related papers  (2020-09-03T19:59:51Z)
• The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation [93.17367076148348]
  We investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset. We unveil that a major cause is the inaccurate classification of object proposals. We propose a simple calibration framework to more effectively alleviate classification head bias with a bi-level class balanced sampling approach.
  arXiv  Detail & Related papers  (2020-07-23T12:49:07Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)
• IVFS: Simple and Efficient Feature Selection for High Dimensional Topology Preservation [33.424663018395684]
  We propose a simple and effective feature selection algorithm to enhance sample similarity preservation. The proposed algorithm is able to well preserve the pairwise distances, as well as topological patterns, of the full data.
  arXiv  Detail & Related papers  (2020-04-02T23:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.