PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

• URL: http://arxiv.org/abs/2605.28819v1
• Date: Wed, 27 May 2026 17:59:51 GMT
• Title: PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
• Abstract summary: We argue that PEFT should be assessed through the stability-plasticity dilemma.<n>We introduce PEFT-Arena, a benchmark that measures downstream performance and general capability retention.<n>Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
• Score: 52.693471818837395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.

Related papers

• Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning [28.24632224083753]
  Mechanistic localization bridges mechanistic interpretability and post-training optimization.<n>We propose a forward-looking framework that accurately estimates the post-SFT interpretability state.
  arXiv  Detail & Related papers  (2026-08-25T12:28:55Z)
• Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning [26.480133417682897]
  Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and forgetting.<n>Recent approaches alleviate this issue by isolating task-critical parameters during training.<n>We propose Evolving Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance.
  arXiv  Detail & Related papers  (2026-04-15T15:55:38Z)
• TPV: Parameter Perturbations Through the Lens of Test Prediction Variance [1.9443009938188445]
  Test prediction variance (TPV) is the first-order sensitivity of model outputs to perturbations around a trained solution.<n>TPV is a fully label-free object whose trace form separates the geometry of the trained model from the specific perturbation mechanism.
  arXiv  Detail & Related papers  (2025-12-11T20:04:33Z)
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics [72.00014675808228]
  Instability in Large Language Models evaluation process obscures true learning dynamics.<n>We introduce textbfMaP, a framework that integrates underlineMerging underlineand the underlinePass@k metric.<n>Experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent rankings.
  arXiv  Detail & Related papers  (2025-10-10T11:40:27Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark [95.0484665958967]
  Pre-trained vision models (PVMs) have demonstrated remarkable adaptability across a wide range of downstream vision tasks.<n>As these models scale to billions or even trillions of parameters, conventional full fine-tuning has become increasingly impractical due to its high computational and storage demands.<n> parameter-efficient fine-tuning (PEFT) has emerged as a promising alternative, aiming to achieve performance comparable to full fine-tuning while making minimal adjustments to the model parameters.
  arXiv  Detail & Related papers  (2024-02-03T19:12:20Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)
• Support estimation in high-dimensional heteroscedastic mean regression [2.07180164747172]
  We consider a linear mean regression model with random design and potentially heteroscedastic, heavy-tailed errors.<n>We use a strictly convex, smooth variant of the Huber loss function with tuning parameter depending on the parameters of the problem.<n>For the resulting estimator we show sign-consistency and optimal rates of convergence in the $ell_infty$ norm.
  arXiv  Detail & Related papers  (2020-11-03T09:46:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.