Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models

• URL: http://arxiv.org/abs/2509.03057v1
• Date: Wed, 03 Sep 2025 06:40:25 GMT
• Title: Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models
• Authors: Ming Gong, Yingnan Deng, Nia Qi, Yujun Zou, Zhihao Xue, Yun Zi,
• Abstract summary: The paper proposes an adapter-based fine-tuning method built on a structure-learnable mechanism.<n>It allows the model to adjust its structure flexibly in multi-task settings to match different task characteristics.<n>It achieves a better balance among accuracy, compression rate, and robustness to noise and perturbation.
• Score: 5.019928514737434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses the issues of parameter redundancy, rigid structure, and limited task adaptability in the fine-tuning of large language models. It proposes an adapter-based fine-tuning method built on a structure-learnable mechanism. By introducing differentiable gating functions and structural sparsity control variables, the method enables automatic optimization of adapter insertion points, activation paths, and module combinations. This allows the model to adjust its structure flexibly in multi-task settings to match different task characteristics. With the backbone parameters kept frozen, the method uses a structure search mechanism to guide the dynamic construction of task-specific efficient substructures during training. This significantly improves parameter utilization and representational capacity. In addition, the paper designs a set of sensitivity analysis experiments to systematically evaluate the effects of sparsity weight, noise injection ratio, and data perturbation on model performance. These experiments verify the stability and robustness of the proposed method across various multi-task natural language understanding tasks. The experimental results show that the proposed method outperforms mainstream parameter-efficient tuning techniques on multiple tasks. It achieves a better balance among accuracy, compression rate, and robustness to noise and perturbation.

Related papers

• GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers [12.475144734899674]
  We introduce GRASP, a lightweight PEFT framework that partitions the D-dimensional token representations of selected layers into K D groups and learns a shared scaling and shifting vector for each group.<n>We propose StochGRASP, which learns Gaussian distributions as perturbations to the pre-trained weights rather than deterministic values.<n>Under varying levels of noise, StochGRASP consistently outperforms deterministic variants, demonstrating its suitability for energy-efficient and noise-prone hardware platforms.
  arXiv  Detail & Related papers  (2025-12-03T22:17:05Z)
• From Coefficients to Directions: Rethinking Model Merging with Directional Alignment [66.99062575537555]
  We introduce a unified geometric framework, emphMerging with Directional Alignment (method), which aligns directional structures consistently in both the parameter and feature spaces.<n>Our analysis shows that directional alignment improves structural coherence, and extensive experiments across benchmarks, model scales, and task configurations further validate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-11-29T08:40:58Z)
• Structural Priors and Modular Adapters in the Composable Fine-Tuning Algorithm of Large-Scale Models [8.15127799301814]
  This paper proposes a composable fine-tuning method that integrates graph structural priors with modular adapters.<n>The proposed framework significantly enhances task prediction accuracy, adapter weight allocation precision, and overall computational efficiency.
  arXiv  Detail & Related papers  (2025-11-06T02:10:20Z)
• Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarios [54.58186816693791]
  environments constantly change over time and space, posing significant challenges for object detectors trained based on a closed-set assumption.<n>We propose a new mechanism, converting the fine-tuning process to a specific- parameter generation.<n>In particular, we first design a dual-path LoRA-based domain-aware adapter that disentangles features into domain-invariant and domain-specific components.
  arXiv  Detail & Related papers  (2025-06-30T17:14:12Z)
• Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution [88.20464308588889]
  We propose a Structural Similarity-Inspired Unfolding (SSIU) method for efficient image SR.<n>This method is designed through unfolding an SR optimization function constrained by structural similarity.<n>Our model outperforms current state-of-the-art models, boasting lower parameter counts and reduced memory consumption.
  arXiv  Detail & Related papers  (2025-06-13T14:29:40Z)
• Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning [5.69600290598441]
  The goal is to improve both fine-tuning efficiency and structural awareness during training.<n>A weighted graph is constructed, and Laplacian spectral decomposition is applied to enable frequency-domain modeling.<n>A spectral filtering mechanism is introduced during the optimization phase, enhancing the model's training stability and convergence behavior.
  arXiv  Detail & Related papers  (2025-04-28T08:42:35Z)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation [67.13876021157887]
  Dynamic Tuning (DyT) is a novel approach to improve both parameter and inference efficiency for ViT adaptation. DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.
  arXiv  Detail & Related papers  (2024-03-18T14:05:52Z)
• Prompt Guided Transformer for Multi-Task Dense Prediction [14.815576352301322]
  We introduce a lightweight task-conditional model called Prompt Guided Transformer to optimize performance and model parameters. Our approach achieves state-of-the-art results among task-conditional methods while using fewer parameters, and maintains a significant balance between performance and parameter size.
  arXiv  Detail & Related papers  (2023-07-28T07:25:57Z)
• Rethinking Efficient Tuning Methods from a Unified Perspective [34.67645496324432]
  We revisit the design paradigm of PETL and derive a unified framework U-Tuning for parameter-efficient transfer learning. The U-Tuning framework can simultaneously encompass existing methods and derive new approaches for parameter-efficient transfer learning.
  arXiv  Detail & Related papers  (2023-03-01T17:38:03Z)
• Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation [11.960178399478718]
  Large pretrained language models (PLMs) are often domain- or task-adapted via fine-tuning or prompting. Instead, we prepare PLMs for data- and parameter-efficient adaptation by learning to learn the difference between general and adapted PLMs.
  arXiv  Detail & Related papers  (2022-07-07T18:00:22Z)
• Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention [53.72897232951918]
  We propose adapters that utilize the kernel structure in self-attention to guide the assignment of tunable parameters. Our results show that our proposed adapters can attain or improve the strong performance of existing baselines.
  arXiv  Detail & Related papers  (2022-05-07T20:52:54Z)
• Towards a Unified View of Parameter-Efficient Transfer Learning [108.94786930869473]
  Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. We break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them.
  arXiv  Detail & Related papers  (2021-10-08T20:22:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.