PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark

• URL: http://arxiv.org/abs/2511.21285v1
• Date: Wed, 26 Nov 2025 11:18:06 GMT
• Title: PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark
• Authors: Robert Belanec, Branislav Pecher, Ivan Srba, Maria Bielikova,
• Abstract summary: We introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs.<n>We demonstrate its usage across 27 NLP datasets and 6 PEFT methods.<n>We also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.
• Score: 8.366144731921489
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the state-of-the-art performance of Large Language Models (LLMs) achieved on many tasks, their massive scale often leads to high computational and environmental costs, limiting their accessibility. Parameter-efficient fine-tuning (PEFT) methods address this challenge by reducing the number of trainable parameters while maintaining strong downstream performance. Despite the increased development in PEFT methods, current evaluations remain limited (in terms of evaluated models and datasets) and difficult to reproduce. To bridge this gap, we introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs. We demonstrate its usage across 27 NLP datasets and 6 PEFT methods. To account for different PEFT training and inference factors, we also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.

Related papers

• PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models [5.63712607768603]
  PEFT-Factory is a unified framework for efficient fine-tuning Large Language Models.<n>It provides a representative set of 19 PEFT methods, 27 classification and text generation datasets, and both standard and PEFT-specific evaluation metrics.
  arXiv  Detail & Related papers  (2025-12-02T13:44:41Z)
• TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating [8.102270371993411]
  We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices.<n>Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive.
  arXiv  Detail & Related papers  (2025-11-20T08:41:20Z)
• Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning [50.05207363001145]
  Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT)<n>We compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory.<n>Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories.
  arXiv  Detail & Related papers  (2025-05-28T13:35:12Z)
• BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models [63.52035708182815]
  We introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT) BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT for downstream tasks with a low parameter budget.
  arXiv  Detail & Related papers  (2024-10-04T18:50:46Z)
• Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning [17.032155725171958]
  We propose the Light-PEFT framework, which includes two methods: Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT. Compared to utilizing the PEFT method directly, Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance.
  arXiv  Detail & Related papers  (2024-06-06T07:03:29Z)
• An Empirical Study of Parameter Efficient Fine-tuning on Vision-Language Pre-train Model [33.853380101736306]
  A natural expectation for PEFTs is that the performance of various PEFTs is positively related to the data size and fine-tunable parameter size. We find that such an intuition holds only if the downstream data and task are not consistent with pre-training. For downstream fine-tuning consistent with pre-training, data size no longer affects the performance, while the influence of fine-tunable parameter size is not monotonous.
  arXiv  Detail & Related papers  (2024-03-13T11:33:38Z)
• Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark [95.0484665958967]
  Pre-trained vision models (PVMs) have demonstrated remarkable adaptability across a wide range of downstream vision tasks.<n>As these models scale to billions or even trillions of parameters, conventional full fine-tuning has become increasingly impractical due to its high computational and storage demands.<n> parameter-efficient fine-tuning (PEFT) has emerged as a promising alternative, aiming to achieve performance comparable to full fine-tuning while making minimal adjustments to the model parameters.
  arXiv  Detail & Related papers  (2024-02-03T19:12:20Z)
• From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers [52.199303258423306]
  We propose a novel density loss that encourages higher activation sparsity in pre-trained models. Our proposed method, textbfDEFT, can consistently reduce activation density by up to textbf44.94% on RoBERTa$_mathrmLarge$ and by textbf53.19% (encoder density) and textbf90.60% (decoder density) on Flan-T5$_mathrmXXL$.
  arXiv  Detail & Related papers  (2024-02-02T21:25:46Z)
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment [12.674032145667763]
  We present a comprehensive and systematic review of Efficient Fine-Tuning (PEFT) methods for pretrained language models (PLMs) PEFT offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. We conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency.
  arXiv  Detail & Related papers  (2023-12-19T13:31:24Z)
• ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization [92.36876698509164]
  We present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models.<n>In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x.
  arXiv  Detail & Related papers  (2023-11-22T05:28:59Z)
• LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning [56.88751562302793]
  Low-rank adaption (LoRA) has emerged to fine-tune large language models (LLMs) LoRAPrune is a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner. LoRAPrune achieves a reduction in perplexity by 4.81 on WikiText2 and 3.46 on PTB, while also decreasing memory usage by 52.6%.
  arXiv  Detail & Related papers  (2023-05-28T15:15:48Z)
• AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning [77.61565726647784]
  Motivated by advances in neural architecture search, we propose AutoPEFT for automatic PEFT configuration selection. We show that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs.
  arXiv  Detail & Related papers  (2023-01-28T08:51:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.