TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating

• URL: http://arxiv.org/abs/2511.16147v1
• Date: Thu, 20 Nov 2025 08:41:20 GMT
• Title: TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating
• Authors: Dabiao Ma, Ziming Dai, Zhimin Xin, Shu Wang, Ye Wang, Haojun Fei,
• Abstract summary: We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices.<n>Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive.
• Score: 8.102270371993411
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the field of large models (LMs) for natural language processing (NLP) and computer vision (CV), Parameter-Efficient Fine-Tuning (PEFT) has emerged as a resource-efficient method that modifies a limited number of parameters while keeping the pretrained weights fixed. This paper investigates the traditional PEFT approach, which applies modifications to all position indices, and questions its necessity. We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices, potentially enhancing performance on downstream tasks. Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive. This study offers a fresh perspective on PEFT, advocating for a more targeted approach to modifications and providing a framework for future research to optimize the fine-tuning process for large models.

Related papers

• PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark [8.366144731921489]
  We introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs.<n>We demonstrate its usage across 27 NLP datasets and 6 PEFT methods.<n>We also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.
  arXiv  Detail & Related papers  (2025-11-26T11:18:06Z)
• GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning [51.79350934271497]
  GateRA is a unified framework that introduces token-aware modulation to dynamically adjust the strength of PEFT updates.<n>By incorporating adaptive gating into standard PEFT branches, GateRA enables selective, token-level adaptation.<n> Experiments on multiple commonsense reasoning benchmarks demonstrate that GateRA consistently outperforms or matches prior PEFT methods.
  arXiv  Detail & Related papers  (2025-11-15T17:55:47Z)
• Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Models [28.79157031663951]
  We introduce WeatherPEFT, a novel PEFT framework for Weather Foundation Models (WFMs)<n>Task-Tu Dynamic Prompting (TTu) injects the embedding weights within the encoder to the input tokens of the backbone of the pre-trained via internal and external pattern extraction.<n>Fisher-Guided Adaptive Selection (SFAS) identifies the most task-critical randomness, preserving in pre-trained knowledge, but also stabilizing the selection.<n>We demonstrate the effectiveness and efficiency of WeatherPEFT on three downstream tasks, where existing PEFT methods show significant gaps versus Full-Tuning, and WeatherPEFT performance parity
  arXiv  Detail & Related papers  (2025-09-26T07:54:05Z)
• FISH-Tuning: Enhancing PEFT Methods with Fisher Information [3.9274736061387854]
  FISH Mask is a selection-based PEFT technique that identifies a critical subset of pre-trained parameters using approximate Fisher information.<n>We propose textbfFISH-Tuning, a novel approach that incorporates FISH Mask into such PEFT methods, including LoRA, Adapter, and their variants.
  arXiv  Detail & Related papers  (2025-04-05T04:05:55Z)
• BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models [63.52035708182815]
  We introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT) BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT for downstream tasks with a low parameter budget.
  arXiv  Detail & Related papers  (2024-10-04T18:50:46Z)
• Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models [18.877891285367216]
  We introduce $textID3$, a novel selective PEFT method that calculates parameter importance continually.<n>We analytically show that $textID3$ reduces the number of gradient updates by a factor of two, enhancing computational efficiency.
  arXiv  Detail & Related papers  (2024-08-26T17:58:53Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• An Empirical Study of Parameter Efficient Fine-tuning on Vision-Language Pre-train Model [33.853380101736306]
  A natural expectation for PEFTs is that the performance of various PEFTs is positively related to the data size and fine-tunable parameter size. We find that such an intuition holds only if the downstream data and task are not consistent with pre-training. For downstream fine-tuning consistent with pre-training, data size no longer affects the performance, while the influence of fine-tunable parameter size is not monotonous.
  arXiv  Detail & Related papers  (2024-03-13T11:33:38Z)
• Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark [95.0484665958967]
  Pre-trained vision models (PVMs) have demonstrated remarkable adaptability across a wide range of downstream vision tasks.<n>As these models scale to billions or even trillions of parameters, conventional full fine-tuning has become increasingly impractical due to its high computational and storage demands.<n> parameter-efficient fine-tuning (PEFT) has emerged as a promising alternative, aiming to achieve performance comparable to full fine-tuning while making minimal adjustments to the model parameters.
  arXiv  Detail & Related papers  (2024-02-03T19:12:20Z)
• Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning [91.5113227694443]
  We propose a novel visual. sensuous-aware fine-Tuning (SPT) scheme. SPT allocates trainable parameters to task-specific important positions. Experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods.
  arXiv  Detail & Related papers  (2023-03-15T12:34:24Z)
• AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning [77.61565726647784]
  Motivated by advances in neural architecture search, we propose AutoPEFT for automatic PEFT configuration selection. We show that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs.
  arXiv  Detail & Related papers  (2023-01-28T08:51:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.