PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models

• URL: http://arxiv.org/abs/2512.02764v1
• Date: Tue, 02 Dec 2025 13:44:41 GMT
• Title: PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
• Authors: Robert Belanec, Ivan Srba, Maria Bielikova,
• Abstract summary: PEFT-Factory is a unified framework for efficient fine-tuning Large Language Models.<n>It provides a representative set of 19 PEFT methods, 27 classification and text generation datasets, and both standard and PEFT-specific evaluation metrics.
• Score: 5.63712607768603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). Currently, many newly introduced PEFT methods are challenging to replicate, deploy, or compare with one another. To address this, we introduce PEFT-Factory, a unified framework for efficient fine-tuning LLMs using both off-the-shelf and custom PEFT methods. While its modular design supports extensibility, it natively provides a representative set of 19 PEFT methods, 27 classification and text generation datasets addressing 12 tasks, and both standard and PEFT-specific evaluation metrics. As a result, PEFT-Factory provides a ready-to-use, controlled, and stable environment, improving replicability and benchmarking of PEFT methods. PEFT-Factory is a downstream framework that originates from the popular LLaMA-Factory, and is publicly available at https://github.com/kinit-sk/PEFT-Factory

Related papers

• PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark [8.366144731921489]
  We introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs.<n>We demonstrate its usage across 27 NLP datasets and 6 PEFT methods.<n>We also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.
  arXiv  Detail & Related papers  (2025-11-26T11:18:06Z)
• TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating [8.102270371993411]
  We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices.<n>Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive.
  arXiv  Detail & Related papers  (2025-11-20T08:41:20Z)
• RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use [50.52940111891476]
  Large language models excel at basic reasoning but struggle with tasks that require interaction with external tools.<n>We present RLFactory, a plug-and-play reinforcement learning framework for multi-round tool use.
  arXiv  Detail & Related papers  (2025-08-31T16:47:31Z)
• PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs [8.52711842775914]
  Efficient Fine-Tuning (PEFT) methods have emerged as effective and promising approaches for fine-tuning pre-trained language models.<n>In this paper, we propose a novel approach, PrunePEFT, which formulates the PEFT strategy search as a pruning problem.
  arXiv  Detail & Related papers  (2025-06-09T09:32:58Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning [17.032155725171958]
  We propose the Light-PEFT framework, which includes two methods: Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT. Compared to utilizing the PEFT method directly, Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance.
  arXiv  Detail & Related papers  (2024-06-06T07:03:29Z)
• ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization [92.36876698509164]
  We present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models.<n>In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x.
  arXiv  Detail & Related papers  (2023-11-22T05:28:59Z)
• Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling [42.42235704360381]
  Large language models (LLMs) and vision language models (VLMs) demonstrate excellent performance on a wide range of tasks. These large scales make it impossible to adapt and deploy fully specialized models given a task of interest. In this work, we describe AdaLink as a non-intrusive PEFT technique that achieves competitive performance.
  arXiv  Detail & Related papers  (2023-10-18T16:43:08Z)
• AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning [77.61565726647784]
  Motivated by advances in neural architecture search, we propose AutoPEFT for automatic PEFT configuration selection. We show that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs.
  arXiv  Detail & Related papers  (2023-01-28T08:51:23Z)
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning [112.97430455461097]
  We propose a general PEFT method that tunes a mixture of adaptation modules introduced in each Transformer layer while keeping most of the PLM weights frozen. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.
  arXiv  Detail & Related papers  (2022-10-31T16:23:36Z)
• UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning [64.638804236566]
  We propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup. Remarkably, on the GLUE benchmark, UniPELT consistently achieves 13pt gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups.
  arXiv  Detail & Related papers  (2021-10-14T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.