Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

• URL: http://arxiv.org/abs/2603.01759v1
• Date: Mon, 02 Mar 2026 11:38:18 GMT
• Title: Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning
• Authors: Zichen Tian, Yaoyao Liu, Qianru Sun,
• Abstract summary: Fine-tuning pre-trained models on remote sensing (RS) images is a straightforward solution.<n>Existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer.<n>We propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning.
• Score: 34.310926877797584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.

Related papers

• TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating [8.102270371993411]
  We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices.<n>Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive.
  arXiv  Detail & Related papers  (2025-11-20T08:41:20Z)
• PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs [8.52711842775914]
  Efficient Fine-Tuning (PEFT) methods have emerged as effective and promising approaches for fine-tuning pre-trained language models.<n>In this paper, we propose a novel approach, PrunePEFT, which formulates the PEFT strategy search as a pruning problem.
  arXiv  Detail & Related papers  (2025-06-09T09:32:58Z)
• PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection [9.152388341742048]
  Fine-tuning large foundation models is essential for building expert models tailored to specialized tasks and domains.<n>We introduce efficient Fine-tuning with Column Space Projection (PiCa), a novel theoretically grounded PEFT method.<n>PiCa consistently outperforms state-of-the-art baselines under comparable or smaller parameter budgets.
  arXiv  Detail & Related papers  (2025-05-26T16:52:40Z)
• CALLIC: Content Adaptive Learning for Lossless Image Compression [64.47244912937204]
  CALLIC sets a new state-of-the-art (SOTA) for learned lossless image compression.<n>We propose a content-aware autoregressive self-attention mechanism by leveraging convolutional gating operations.<n>During encoding, we decompose pre-trained layers, including depth-wise convolutions, using low-rank matrices and then adapt the incremental weights on testing image by Rate-guided Progressive Fine-Tuning (RPFT)<n>RPFT fine-tunes with gradually increasing patches that are sorted in descending order by estimated entropy, optimizing learning process and reducing adaptation time.
  arXiv  Detail & Related papers  (2024-12-23T10:41:18Z)
• Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models [24.62337386603331]
  Large Multi-modal Models (LMMs) are revolutionizing the way machines interact with the world. To adapt LMMs for downstream tasks, parameter-efficient fine-tuning (PEFT) has gained popularity. This paper focuses on the strengths and weaknesses of each tuning strategy, shifting the focus from the efficiency typically associated with these approaches.
  arXiv  Detail & Related papers  (2024-10-29T07:55:50Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• Context-PEFT: Efficient Multi-Modal, Multi-Task Fine-Tuning [12.648711621637663]
  This paper introduces a novel. COCO-Efficient Fine-Tuning (PEFT) framework for multi-modal, multi-task transfer learning with pre-trained language models. We propose Context-PEFT, which learns different groups of adaptor parameters based on the token's domain. Our method is evaluated on the captioning task, where it outperforms full fine-tuning under similar data constraints.
  arXiv  Detail & Related papers  (2023-12-14T13:00:24Z)
• PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for Infrared Images [45.507517332100804]
  Self-supervised learning (SSL) for RGB images has achieved significant success, yet there is still limited research on SSL for infrared images. Non-iconic infrared images rendering common pre-training tasks less effective. The scarcity of fine-grained textures making it particularly challenging to learn general image features.
  arXiv  Detail & Related papers  (2023-12-13T14:57:28Z)
• Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts [52.39959535724677]
  We introduce an alternative solution to improve the generalization of image restoration models. We propose AdaptIR, a Mixture-of-Experts (MoE) with multi-branch design to capture local, global, and channel representation bases. Our AdaptIR achieves stable performance on single-degradation tasks, and excels in hybrid-degradation tasks, with fine-tuning only 0.6% parameters for 8 hours.
  arXiv  Detail & Related papers  (2023-12-12T14:27:59Z)
• DGNet: Dynamic Gradient-Guided Network for Water-Related Optics Image Enhancement [77.0360085530701]
  Underwater image enhancement (UIE) is a challenging task due to the complex degradation caused by underwater environments. Previous methods often idealize the degradation process, and neglect the impact of medium noise and object motion on the distribution of image features. Our approach utilizes predicted images to dynamically update pseudo-labels, adding a dynamic gradient to optimize the network's gradient space.
  arXiv  Detail & Related papers  (2023-12-12T06:07:21Z)
• Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model [81.55141188169621]
  We equip PEFT with a cross-block orchestration mechanism to enable the adaptation of the Segment Anything Model (SAM) to various downstream scenarios. We propose an intra-block enhancement module, which introduces a linear projection head whose weights are generated from a hyper-complex layer. Our proposed approach consistently improves the segmentation performance significantly on novel scenarios with only around 1K additional parameters.
  arXiv  Detail & Related papers  (2023-11-28T11:23:34Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)
• AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning [77.61565726647784]
  Motivated by advances in neural architecture search, we propose AutoPEFT for automatic PEFT configuration selection. We show that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs.
  arXiv  Detail & Related papers  (2023-01-28T08:51:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.