FACT: A Simple and Efficient Framework for Active Finetuning

• URL: http://arxiv.org/abs/2606.02079v1
• Date: Mon, 01 Jun 2026 11:08:31 GMT
• Title: FACT: A Simple and Efficient Framework for Active Finetuning
• Abstract summary: We propose FACT, a three-phase hierarchical finetuning framework featuring both efficiency and simplicity.<n>Under low sampling ratios, our framework achieves remarkable performance gains of over 20% on the ViT model for CIFAR10, CIFAR100, and ImageNet-1k benchmarks.<n>This systematic approach establishes new state-of-the-art performance while maintaining parameter efficiency, proving particularly effective when labeled data is scarce.
• Score: 23.983494767170644
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The main goal of active finetuning is to improve a pretrained model's performance on a specific task or domain by finetuning it with carefully selected informative or challenging data. Previous research has predominantly focused on the active aspect (i.e., data selection) while uniformly employing full finetuning for model adaptation, which inevitably distorts pretrained features due to distribution shift. This issue becomes particularly pronounced when the model size is large relative to the finetuning data quantity, leading to heightened overfitting risks. To address this critical gap, we formally outline the FiAF task that emphasizes systematic exploration of finetuning methodologies in active learning. We propose FACT, a three-phase hierarchical finetuning framework featuring both efficiency and simplicity, specifically designed for active finetuning scenarios. Our comprehensive experiments span: (1) Three major dataset categories encompassing classic (CIFAR10, CIFAR100, ImageNet-1k), imbalanced (CIFAR10-LT, CIFAR100-LT), and fine-grained (StanfordCars, FGVCAircraft) image classification datasets, each evaluated under 3-5 distinct sampling ratios; (2) Diverse pretrained architectures including Convolutional Neural Network (ConvNeXt), Vision Transformer (ViT), and Vision LSTM (ViL) networks; (3) A systematic investigation of frozen feature augmentation (FroFA) strategies. (4) A comprehensive and rigorous analysis of efficiency and generalizability. The results demonstrate significant improvements with strong generalization and robustness. Notably, under low sampling ratios, our framework achieves remarkable performance gains of over 20% on the ViT model for CIFAR10, CIFAR100, and ImageNet-1k benchmarks. This systematic approach establishes new state-of-the-art performance while maintaining parameter efficiency, proving particularly effective when labeled data is scarce.

Related papers

• Efficient Adversarial Training via Criticality-Aware Fine-Tuning [59.84340722595335]
  Criticality-Aware Adversarial Training (CAAT) is a novel method that adaptively allocates resources to the most robustness-critical parameters.<n>CAAT incurs only a 4.3% decrease in adversarial robustness while tuning approximately 6% of its parameters.<n>Experiments on three widely used adversarial learning datasets demonstrate that CAAT outperforms state-of-the-art lightweight AT methods with fewer trainable parameters.
  arXiv  Detail & Related papers  (2026-04-14T14:17:38Z)
• Adapting General-Purpose Foundation Models for X-ray Ptychography in Low-Data Regimes [8.748610895973075]
  We introduce PtychoBench, a new benchmark for ptychographic analysis.<n>We compare two specialization strategies: Supervised Fine-Tuning (SFT) and In-Context Learning (ICL)<n>Our findings reveal that the optimal specialization pathway is task-dependent.
  arXiv  Detail & Related papers  (2025-11-04T11:43:05Z)
• 15,500 Seconds: Lean UAV Classification Using EfficientNet and Lightweight Fine-Tuning [2.3354223046061016]
  This paper addresses the challenge of data scarcity in UAV audio classification by expanding on prior work.<n>We use a custom dataset of 3,100 UAV audio clips (15,500 seconds) spanning 31 distinct drone types.<n>Experiments were conducted with five-fold cross-validation, assessing accuracy, training efficiency, and robustness.
  arXiv  Detail & Related papers  (2025-05-21T21:53:19Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness [56.87603097348203]
  VeCAF uses labels and natural language annotations to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning.
  arXiv  Detail & Related papers  (2024-01-15T17:28:37Z)
• E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning [55.50908600818483]
  Fine-tuning large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. We propose an Effective and Efficient Visual Prompt Tuning (E2VPT) approach for large-scale transformer-based model adaptation. Our approach outperforms several state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T19:03:21Z)
• T-ADAF: Adaptive Data Augmentation Framework for Image Classification Network based on Tensor T-product Operator [0.0]
  This paper proposes an Adaptive Data Augmentation Framework based on the tensor T-product Operator. It triples one image data to be trained and gain the result from all these three images together with only less than 0.1% increase in the number of parameters. Numerical experiments show that our data augmentation framework can improve the performance of original neural network model by 2%.
  arXiv  Detail & Related papers  (2023-06-07T08:30:44Z)
• GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer [76.2625311630021]
  Vision transformers (ViTs) have shown very impressive empirical performance in various computer vision tasks. To mitigate this challenging problem, structured pruning is a promising solution to compress model size and enable practical efficiency. We propose GOHSP, a unified framework of Graph and Optimization-based Structured Pruning for ViT models.
  arXiv  Detail & Related papers  (2023-01-13T00:40:24Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• Stochastic Optimization of Plain Convolutional Neural Networks with Simple methods [0.0]
  We present a combination of regularization techniques which work together to get better performance. We tested and evaluated these techniques by applying models on five famous datasets.
  arXiv  Detail & Related papers  (2020-01-24T01:20:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.