PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

• URL: http://arxiv.org/abs/2504.14117v1
• Date: Sat, 19 Apr 2025 00:33:16 GMT
• Title: PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
• Authors: Nusrat Jahan Prottasha, Upama Roy Chowdhury, Shetu Mohanto, Tasfia Nuzhat, Abdullah As Sami, Md Shamol Ali, Md Shohanur Islam Sobuj, Hafijur Raman, Md Kowsher, Ozlem Ozmen Garibay,
• Abstract summary: Large models such as Large Language Models (LLMs) and Vision Language Models (VLMs) have transformed artificial intelligence.<n>Fine-tuning these models remains expensive, requiring extensive computational resources, memory, and task-specific data.<n>Efficient Fine-Tuning (PEFT) has emerged as a promising solution that allows adapting large models to downstream tasks by updating only a small portion of parameters.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large models such as Large Language Models (LLMs) and Vision Language Models (VLMs) have transformed artificial intelligence, powering applications in natural language processing, computer vision, and multimodal learning. However, fully fine-tuning these models remains expensive, requiring extensive computational resources, memory, and task-specific data. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a promising solution that allows adapting large models to downstream tasks by updating only a small portion of parameters. This survey presents a comprehensive overview of PEFT techniques, focusing on their motivations, design principles, and effectiveness. We begin by analyzing the resource and accessibility challenges posed by traditional fine-tuning and highlight key issues, such as overfitting, catastrophic forgetting, and parameter inefficiency. We then introduce a structured taxonomy of PEFT methods -- grouped into additive, selective, reparameterized, hybrid, and unified frameworks -- and systematically compare their mechanisms and trade-offs. Beyond taxonomy, we explore the impact of PEFT across diverse domains, including language, vision, and generative modeling, showing how these techniques offer strong performance with lower resource costs. We also discuss important open challenges in scalability, interpretability, and robustness, and suggest future directions such as federated learning, domain adaptation, and theoretical grounding. Our goal is to provide a unified understanding of PEFT and its growing role in enabling practical, efficient, and sustainable use of large models.

Related papers

• A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning [5.280048850098648]
  Foundation models have revolutionized artificial intelligence by providing robust, versatile architectures pre-trained on large-scale datasets. Adapting these massive models to specific downstream tasks requires fine-tuning, which can be prohibitively expensive in computational resources. This survey provides a comprehensive review of the integration of PEFT techniques within federated learning environments.
  arXiv  Detail & Related papers  (2025-04-29T18:18:39Z)
• A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models [2.171120568435925]
  Large Language Models (LLMs) for code require significant computational resources for training and fine-tuning. To address this, the research community has increasingly turned to Efficient Fine-Tuning (PEFT) PEFT enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model. Our study synthesizes findings from 27 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs.
  arXiv  Detail & Related papers  (2025-04-29T16:19:25Z)
• Fine-tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models [16.522696273752835]
  Earth observation is crucial for monitoring environmental changes, responding to disasters, and managing natural resources. Foundation models facilitate remote sensing image analysis to retrieve relevant geoinformation accurately and efficiently. As these models grow in size, fine-tuning becomes increasingly challenging due to associated computational resources and costs.
  arXiv  Detail & Related papers  (2025-04-24T09:37:02Z)
• Unified Parameter-Efficient Unlearning for LLMs [25.195126838721492]
  Large Language Models (LLMs) have revolutionized natural language processing, enabling advanced understanding and reasoning capabilities across a variety of tasks.<n>This raises significant privacy and security concerns, as models may inadvertently retain and disseminate sensitive or undesirable information.<n>We introduce a novel instance-wise unlearning framework, LLMEraser, which systematically categorizes unlearning tasks and applies precise adjustments using influence functions.
  arXiv  Detail & Related papers  (2024-11-30T07:21:02Z)
• Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies [17.904502959675337]
  Large models require substantial computational resources and GPU memory to operate. Efficient Fine-Tuning (PEFT) offers a practical solution by efficiently adjusting the parameters of large pre-trained models to suit various downstream tasks. This review mainly introduces the preliminary knowledge of PEFT, the core ideas and principles of various PEFT algorithms, the applications of PEFT, and potential future research directions.
  arXiv  Detail & Related papers  (2024-10-24T13:58:59Z)
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey [18.00772798876708]
  Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain. We present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead.
  arXiv  Detail & Related papers  (2024-03-21T17:55:50Z)
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment [12.674032145667763]
  We present a comprehensive and systematic review of Efficient Fine-Tuning (PEFT) methods for pretrained language models (PLMs) PEFT offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. We conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency.
  arXiv  Detail & Related papers  (2023-12-19T13:31:24Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression [64.07696663255155]
  Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. We introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT) which effectively transfers the knowledge of LLMs to extremely small-scale models.
  arXiv  Detail & Related papers  (2023-10-24T07:58:20Z)
• Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
  We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
  arXiv  Detail & Related papers  (2023-03-13T16:00:31Z)
• Differentiable modeling to unify machine learning and physical models and advance Geosciences [38.92849886903847]
  We outline the concepts, applicability, and significance of differentiable geoscientific modeling (DG) "Differentiable" refers to accurately and efficiently calculating gradients with respect to model variables. Preliminary evidence suggests DG offers better interpretability and causality than Machine Learning.
  arXiv  Detail & Related papers  (2023-01-10T15:24:14Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Pre-Trained Models: Past, Present and Future [126.21572378910746]
  Large-scale pre-trained models (PTMs) have recently achieved great success and become a milestone in the field of artificial intelligence (AI) By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch.
  arXiv  Detail & Related papers  (2021-06-14T02:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.