A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

• URL: http://arxiv.org/abs/2504.21569v1
• Date: Tue, 29 Apr 2025 16:19:25 GMT
• Title: A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
• Authors: Md Zahidul Haque, Saima Afrin, Antonio Mastropaolo,
• Abstract summary: Large Language Models (LLMs) for code require significant computational resources for training and fine-tuning.<n>To address this, the research community has increasingly turned to Efficient Fine-Tuning (PEFT)<n>PEFT enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model.<n>Our study synthesizes findings from 27 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs.
• Score: 2.171120568435925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rise of Artificial Intelligence (AI)-and particularly Large Language Models (LLMs) for code-has reshaped Software Engineering (SE) by enabling the automation of tasks such as code generation, bug detection, and repair. However, these models require significant computational resources for training and fine-tuning, posing challenges for real-world adoption in resource-constrained environments. To address this, the research community has increasingly turned to Parameter-Efficient Fine-Tuning (PEFT)-a class of techniques that enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model. In this Systematic Literature Review (SLR), we examine the growing application of PEFT techniques-across a wide range of software engineering tasks. We analyze how these methods are used to optimize various deep learning (DL) architectures, focusing on their impact on both performance and efficiency. Our study synthesizes findings from 27 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs. The outcome of this review is a comprehensive taxonomy that categorizes PEFT usage by task type, distinguishing between generative (e.g., Code Summarization) and non-generative (e.g., Code Clone Detection) scenarios. Our findings aim to inform future research and guide the practical deployment of PEFT in sustainable, AI-powered software development. Our artifacts are publicly available at https://github.com/alvi75/SLR-PEFT

Related papers

• A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning [5.280048850098648]
  Foundation models have revolutionized artificial intelligence by providing robust, versatile architectures pre-trained on large-scale datasets.<n>Adapting these massive models to specific downstream tasks requires fine-tuning, which can be prohibitively expensive in computational resources.<n>This survey provides a comprehensive review of the integration of PEFT techniques within federated learning environments.
  arXiv  Detail & Related papers  (2025-04-29T18:18:39Z)
• Fine-tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models [16.522696273752835]
  Earth observation is crucial for monitoring environmental changes, responding to disasters, and managing natural resources.<n>Foundation models facilitate remote sensing image analysis to retrieve relevant geoinformation accurately and efficiently.<n>As these models grow in size, fine-tuning becomes increasingly challenging due to associated computational resources and costs.
  arXiv  Detail & Related papers  (2025-04-24T09:37:02Z)
• PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models [0.0]
  Large models such as Large Language Models (LLMs) and Vision Language Models (VLMs) have transformed artificial intelligence.<n>Fine-tuning these models remains expensive, requiring extensive computational resources, memory, and task-specific data.<n>Efficient Fine-Tuning (PEFT) has emerged as a promising solution that allows adapting large models to downstream tasks by updating only a small portion of parameters.
  arXiv  Detail & Related papers  (2025-04-19T00:33:16Z)
• Parameter-Efficient Continual Fine-Tuning: A Survey [5.59258786465086]
  We believe the next breakthrough in AI lies in enabling efficient adaptation to evolving environments.<n>One alternative to efficiently adapt these large-scale models is known.<n>Efficient Fine-Tuning (PEFT)
  arXiv  Detail & Related papers  (2025-04-18T17:51:51Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models. Our framework incorporates two complementary strategies: internal TTC and external TTC. We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• A Survey of Small Language Models [104.80308007044634]
  Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources. We present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques.
  arXiv  Detail & Related papers  (2024-10-25T23:52:28Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Inference Optimization of Foundation Models on AI Accelerators [68.24450520773688]
  Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI. As the number of model parameters reaches to hundreds of billions, their deployment incurs prohibitive inference costs and high latency in real-world scenarios. This tutorial offers a comprehensive discussion on complementary inference optimization techniques using AI accelerators.
  arXiv  Detail & Related papers  (2024-07-12T09:24:34Z)
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey [18.00772798876708]
  Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain. We present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead.
  arXiv  Detail & Related papers  (2024-03-21T17:55:50Z)
• The Efficiency Spectrum of Large Language Models: An Algorithmic Survey [54.19942426544731]
  The rapid growth of Large Language Models (LLMs) has been a driving force in transforming various domains. This paper examines the multi-faceted dimensions of efficiency essential for the end-to-end algorithmic development of LLMs.
  arXiv  Detail & Related papers  (2023-12-01T16:00:25Z)
• Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models [11.845239346943067]
  parameter-efficient fine-tuning (PEFT) is a promising approach to efficiently specialize large language models (LLMs) to task-specific data.<n>Our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization.
  arXiv  Detail & Related papers  (2023-08-21T04:31:06Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.