CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation

• URL: http://arxiv.org/abs/2305.04526v2
• Date: Sun, 9 Jul 2023 00:08:11 GMT
• Title: CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation
• Abstract summary: Post-training compression techniques such as pruning and quantization can help lower deployment costs. We propose CrAFT, a simple fine-tuning framework that enables effective post-training network compression. The CrAFT approach adds negligible training overhead as fine-tuning is done in under a couple of minutes or hours with a single GPU.
• Score: 3.043665249713003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transfer learning has become a popular task adaptation method in the era of foundation models. However, many foundation models require large storage and computing resources, which makes off-the-shelf deployment impractical. Post-training compression techniques such as pruning and quantization can help lower deployment costs. Unfortunately, the resulting performance degradation limits the usability and benefits of such techniques. To close this performance gap, we propose CrAFT, a simple fine-tuning framework that enables effective post-training network compression. In CrAFT, users simply employ the default fine-tuning schedule along with sharpness minimization objective, simultaneously facilitating task adaptation and compression-friendliness. Contrary to the conventional sharpness minimization techniques, which are applied during pretraining, the CrAFT approach adds negligible training overhead as fine-tuning is done in under a couple of minutes or hours with a single GPU. The effectiveness of CrAFT, which is a general-purpose tool that can significantly boost one-shot pruning and post-training quantization, is demonstrated on both convolution-based and attention-based vision foundation models on a variety of target tasks. The code will be made publicly available.

Related papers

• Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performance [47.605498477489306]
  Some advanced finetuning methods with auxiliary training objectives can improve performance and reduce the number of convergence steps.<n>We propose a novel approach to address the challenge that pretrained VLA models often fail to effectively improve performance and reduce adaptation costs during standard supervised finetuning.
  arXiv  Detail & Related papers  (2026-03-26T17:14:57Z)
• Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog [72.4168434368873]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources.<n>We propose a gradual compacting method that divides the compression process into multiple fine-grained iterations.<n>This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss.
  arXiv  Detail & Related papers  (2026-02-04T06:56:52Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• C-SWAP: Explainability-Aware Structured Pruning for Efficient Neural Networks Compression [4.10373648742522]
  Pruning is a widely used technique that prompts sparsity in model structures.<n>We propose a novel one-shot pruning framework that relies on explainable deep learning.<n>Our method consistently achieves substantial reductions in model size, with minimal impact on performance, and without the need for fine-tuning.
  arXiv  Detail & Related papers  (2025-10-21T13:40:11Z)
• Efficient Edge Test-Time Adaptation via Latent Feature Coordinate Correction [43.48832321879385]
  We propose a novel test-time adaptation (TTA) method tailored for edge devices (TED)<n>TED employs forward-only coordinate optimization in the principal subspace of latent using the covariance matrix adaptation evolution strategy (CMA-ES)<n>TED achieves state-of-the-art performance while $textitreducing computational complexity by up to 63 times$, offering a practical and scalable solution for real-world edge applications.
  arXiv  Detail & Related papers  (2025-10-13T07:08:52Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved [59.79302182800274]
  Token compression is essential for reducing the computational and memory requirements of transformer models. We propose an efficient and hardware-compatible token compression method called Prune and Merge.
  arXiv  Detail & Related papers  (2025-03-30T14:23:18Z)
• Choose Your Model Size: Any Compression by a Single Gradient Descent [9.074689052563878]
  We present Any Compression via Iterative Pruning (ACIP) ACIP is an algorithmic approach to determine a compression-performance trade-off from a single gradient descent run. We show that ACIP seamlessly complements common quantization-based compression techniques.
  arXiv  Detail & Related papers  (2025-02-03T18:40:58Z)
• Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning [63.43972993473501]
  Token compression expedites the training and inference of Vision Transformers (ViTs) However, when applied to downstream tasks, compression degrees are mismatched between training and inference stages. We propose a model arithmetic framework to decouple the compression degrees between the two stages.
  arXiv  Detail & Related papers  (2024-08-13T10:36:43Z)
• PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation [61.57833648734164]
  We propose a novel Parallel Yielding Re-Activation (PYRA) method for training-inference efficient task adaptation. PYRA outperforms all competing methods under both low compression rate and high compression rate.
  arXiv  Detail & Related papers  (2024-03-14T09:06:49Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees [53.950234267704]
  We introduce Global-QSGD, an All-reduce gradient-compatible quantization method.<n>We show that it accelerates distributed training by up to 3.51% over baseline quantization methods.
  arXiv  Detail & Related papers  (2023-05-29T21:32:15Z)
• Just CHOP: Embarrassingly Simple LLM Compression [27.64461490974072]
  Large language models (LLMs) enable unparalleled few- and zero-shot reasoning capabilities but at a high computational footprint. We show that simple layer pruning coupled with an extended language model pretraining produces state-of-the-art results against structured and even semi-structured compression of models at a 7B scale. We also show how distillation, which has been super effective in task-agnostic compression of smaller BERT-style models, becomes inefficient against our simple pruning technique.
  arXiv  Detail & Related papers  (2023-05-24T08:18:35Z)
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning [29.284147465251685]
  We introduce a new compression framework which covers both weight pruning and quantization in a unified setting. We show that it can improve significantly upon the compression-accuracy trade-offs of existing post-training methods.
  arXiv  Detail & Related papers  (2022-08-24T14:33:35Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Practical Network Acceleration with Tiny Sets [38.742142493108744]
  Network compression is effective in accelerating the inference of deep neural networks. But it often requires finetuning with all the training data to recover from the accuracy loss. We propose a method named PRACTISE to accelerate the network with tiny sets of training images.
  arXiv  Detail & Related papers  (2022-02-16T05:04:38Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [65.68511423300812]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. ProgFed inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)
• You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient [88.58536093633167]
  Existing model compression approaches require re-compression or fine-tuning across diverse constraints to accommodate various hardware deployments. We propose a novel approach, YOCO-BERT, to achieve compress once and deploy everywhere. Compared with state-of-the-art algorithms, YOCO-BERT provides more compact models, yet achieving 2.1%-4.5% average accuracy improvement on the GLUE benchmark.
  arXiv  Detail & Related papers  (2021-06-04T12:17:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.