ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

• URL: http://arxiv.org/abs/2604.19254v1
• Date: Tue, 21 Apr 2026 09:17:35 GMT
• Title: ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
• Abstract summary: Low-Rank Adaptation (LoRA) achieves adaptation by inserting independent low-rank perturbations directly to individual weights.<n>We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module.
• Score: 14.501555924276824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

Related papers

• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs [18.457250148137494]
  We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained large language models into depth-sparse models without retraining.<n>On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts.
  arXiv  Detail & Related papers  (2026-07-23T13:13:04Z)
• Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understanding [4.750456629699177]
  Point Ladder Tuning (PLT) is a PEFT framework that performs hierarchical, instance-conditioned adaptation while keeping the backbone frozen.<n>For dense prediction, we introduce a lightweight segmentation head that progressively upsamples fused features.<n>Experiments on classification and dense prediction show that consistently surpasses prior PEFT baselines with minimal parameters.
  arXiv  Detail & Related papers  (2026-07-21T15:06:37Z)
• Small Data, Big Noise: Adversarial Training for Robust Parameter-Efficient Fine-Tuning [3.0344469521198003]
  We propose a unified framework that brings adversarial training to PEFT.<n>This framework addresses the less explored intersection of adversarial training and parameter-efficient adaptation.<n>It makes PEFT deployments more reliable in real-world scenarios where data scarcity and linguistic variability often coexist.
  arXiv  Detail & Related papers  (2026-06-09T09:11:33Z)
• Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT [12.18095306054314]
  Conservative Supervised Fine-Tuning (ConSFT) is an optimization objective that adapts to target distributions while mitigating catastrophic forgetting.<n>We evaluate ConSFT on the LIBERO and RoboTwin benchmarks across state-of-the-art flow-matching VLAs.
  arXiv  Detail & Related papers  (2026-05-09T10:59:03Z)
• Regularizing Subspace Redundancy of Low-Rank Adaptation [54.473090597164834]
  We propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation.<n>Our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks.<n>As a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs.
  arXiv  Detail & Related papers  (2025-07-28T11:52:56Z)
• PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection [9.152388341742048]
  Fine-tuning large foundation models is essential for building expert models tailored to specialized tasks and domains.<n>We introduce efficient Fine-tuning with Column Space Projection (PiCa), a novel theoretically grounded PEFT method.<n>PiCa consistently outperforms state-of-the-art baselines under comparable or smaller parameter budgets.
  arXiv  Detail & Related papers  (2025-05-26T16:52:40Z)
• MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning [5.412348391086257]
  We propose MSPLoRA, which introduces Global Shared LoRA, Mid-Level Shared LoRA, and Layer-Specific LoRA to capture global patterns, mid-level features, and fine-grained information.<n> Experiments on various NLP tasks demonstrate that MSPLoRA achieves more efficient adaptation and better performance while significantly reducing the number of trainable parameters.
  arXiv  Detail & Related papers  (2025-03-27T07:01:50Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• ConvLoRA and AdaBN based Domain Adaptation via Self-Training [4.006331916849688]
  We propose Convolutional Low-Rank Adaptation (ConvLoRA) for multi-target domain adaptation. ConvLoRA freezes pre-trained model weights, adds trainable low-rank decomposition matrices to convolutional layers, and backpropagates the gradient. Our method has fewer trainable parameters and performs better or on-par with large independent fine-tuned networks.
  arXiv  Detail & Related papers  (2024-02-07T15:43:50Z)
• Dynamic Layer Tying for Parameter-Efficient Transformers [65.268245109828]
  We employ Reinforcement Learning to select layers during training and tie them together. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method.
  arXiv  Detail & Related papers  (2024-01-23T14:53:20Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Spectral Tensor Train Parameterization of Deep Learning Layers [136.4761580842396]
  We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. We show the effects of neural network compression in the classification setting and both compression and improved stability training in the generative adversarial training setting.
  arXiv  Detail & Related papers  (2021-03-07T00:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.