Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation

• URL: http://arxiv.org/abs/2608.08336v1
• Date: Sat, 08 Aug 2026 21:04:19 GMT
• Title: Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation
• Abstract summary: Circuit Fine-Tuning (CFT) is a compute-efficient framework that uses circuit discovery to select modules for fine-tuning before training.<n>CFT needs no learning-rate warmup and reaches peak accuracy in $sim20$ epochs on average.
• Score: 3.52359746858894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has become the de facto standard for adapting Vision Transformers (ViTs) to downstream tasks. While parameter count has been the dominant efficiency metric in PEFT, it does not imply \textit{compute efficiency}: parameter-sparse methods can still incur full-model training cost per step, and typically need long schedules to reach peak accuracy. We introduce Circuit Fine-Tuning (CFT), a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training. Whereas attribution is conventionally formulated against a trained task head, we formulate it against a near-zero-initialized probe head, which isolates the response of the backbone to the target distribution rather than the preferences of a particular classifier. CFT then fine-tunes only the recovered subgraph. CFT needs no learning-rate warmup and reaches peak accuracy in ${\sim}20$ epochs on average---versus $44$--$96$ for strong PEFT baselines---yielding $2.3$--$6.6\times$ fewer training FLOPs and up to $16\times$ less wall-clock time, while adding zero parameters and no inference operations. Experiments across a standard visual transfer benchmark (VTAB-1k), hierarchical backbones (Swin), domain-shifted medical imaging (CBIS-DDSM), and a vision-language model (Gemma-3 on CUB-200) demonstrate the effectiveness of CFT. Code is available at https://github.com/UriKialy/CFT

Related papers

• Quantum Tunneling-Aware Machine Learning: Physics-Derived Noise Models for Robust Deployment [5.373588450790602]
  We introduce quantum tunneling-aware machine learning (QTAML)<n>We derive the deployment-time weight-error distribution from first principles using the Wentzel-Kramers-Brillouin (WKB) approximation.<n>We package these structural properties into a single deployment-time algorithm, Tunneling-Aware Compensation (TAC), that combines closed-form mean correction with an optimal layer-budget allocation.
  arXiv  Detail & Related papers  (2026-05-30T14:21:16Z)
• PreFT: Prefill-only finetuning for efficient inference [54.58291801311547]
  We develop and release an efficient implementation of two prefill-only PEFTs, LoRA and ReFT, on the vLLM inference engine.<n>We show that serving multi-user PreFTs is more efficient than traditional PEFTs ($1.9times$ the throughput when serving $512$ adapters on Llama 3.1 70B)
  arXiv  Detail & Related papers  (2026-05-14T00:19:41Z)
• DualTCN: A Physics-Constrained Temporal Convolutional Network for 2 Time-Domain Marine CSEM Inversion [1.2031796234206138]
  DualTCN is the first deep-learning framework for inverting time-domain marine controlled-source electromagnetic (MCSEM) transient data.<n>The framework regresses four earth-model parameters and reconstructs profiles using a differentiable soft-step decoder.<n>The framework demonstrates high robustness to noise through curriculum-based amplitude augmentation, maintaining a mean $barR2$ of 0.858 at $pm2%$ random amplitude error.
  arXiv  Detail & Related papers  (2026-05-06T14:58:17Z)
• Self-Auditing Parameter-Efficient Fine-Tuning for Few-Shot 3D Medical Image Segmentation [0.0]
  SEA-PEFT treats adapter configuration as an online allocation problem solved during fine-tuning.<n>SEA-PEFT uses a search-audit-al loop that trains active adapters, estimates each adapter's Dice utility by momentarily toggling it off, and then reselects the active set under a parameter budget.<n>On TotalSegmentator and FLARE'22, SEA-PEFT improves mean Dice by 2.4--2.8 points over the strongest fixed-topology PEFT smoothing.
  arXiv  Detail & Related papers  (2026-03-06T02:17:12Z)
• VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models [0.8875650122536799]
  We introduce VectorFit, a new way of parameterization that efficiently utilizes the existing knowledge embedded in $W$ by adaptively training their singular vectors and biases.<n>We show that utilizing the structural and transformational properties of $W$ in this way can lead to high-rank incremental weight matrices $Delta W$, comparable to that of full fine-tuning.
  arXiv  Detail & Related papers  (2025-03-25T10:36:27Z)
• From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers [52.199303258423306]
  We propose a novel density loss that encourages higher activation sparsity in pre-trained models. Our proposed method, textbfDEFT, can consistently reduce activation density by up to textbf44.94% on RoBERTa$_mathrmLarge$ and by textbf53.19% (encoder density) and textbf90.60% (decoder density) on Flan-T5$_mathrmXXL$.
  arXiv  Detail & Related papers  (2024-02-02T21:25:46Z)
• Data-Free Dynamic Compression of CNNs for Tractable Efficiency [46.498278084317704]
  structured pruning approaches have shown promise in lowering floating-point operations without substantial drops in accuracy.<n>We propose HASTE (Hashing for Tractable Efficiency), a data-free, plug-and-play convolution module that instantly reduces a network's test-time inference cost without training or fine-tuning.<n>We demonstrate our approach on the popular vision benchmarks CIFAR-10 and ImageNet, where we achieve a 46.72% reduction in FLOPs with only a 1.25% loss in accuracy.
  arXiv  Detail & Related papers  (2023-09-29T13:09:40Z)
• Does Continual Learning Equally Forget All Parameters? [55.431048995662714]
  Distribution shift (e.g., task or domain shift) in continual learning (CL) usually results in catastrophic forgetting of neural networks. We study which modules in neural networks are more prone to forgetting by investigating their training dynamics during CL. We propose a more efficient and simpler method that entirely removes the every-step replay and replaces them by only $k$-times of FPF periodically triggered during CL.
  arXiv  Detail & Related papers  (2023-04-09T04:36:24Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)
• Improving Representational Continuity via Continued Pretraining [76.29171039601948]
  Transfer learning community (LP-FT) outperforms naive training and other continual learning methods. LP-FT also reduces forgetting in a real world satellite remote sensing dataset (FMoW) variant of LP-FT gets state-of-the-art accuracies on an NLP continual learning benchmark.
  arXiv  Detail & Related papers  (2023-02-26T10:39:38Z)
• Online Hyperparameter Optimization for Class-Incremental Learning [99.70569355681174]
  Class-incremental learning (CIL) aims to train a classification model while the number of classes increases phase-by-phase. An inherent challenge of CIL is the stability-plasticity tradeoff, i.e., CIL models should keep stable to retain old knowledge and keep plastic to absorb new knowledge. We propose an online learning method that can adaptively optimize the tradeoff without knowing the setting as a priori.
  arXiv  Detail & Related papers  (2023-01-11T17:58:51Z)
• Decoder Tuning: Efficient Language Understanding as Decoding [84.68266271483022]
  We present Decoder Tuning (DecT), which in contrast optimize task-specific decoder networks on the output side. By gradient-based optimization, DecT can be trained within several seconds and requires only one P query per sample. We conduct extensive natural language understanding experiments and show that DecT significantly outperforms state-of-the-art algorithms with a $200times$ speed-up.
  arXiv  Detail & Related papers  (2022-12-16T11:15:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.