Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning

• URL: http://arxiv.org/abs/2603.21970v1
• Date: Mon, 23 Mar 2026 13:35:11 GMT
• Title: Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning
• Abstract summary: This paper compares three adaptation approaches-Low-Rank Adaptation (LoRA), Prompt Tuning, and Full Fine-Tuning-across the Flan-T5 model family on the PubMed medical summarization dataset.<n>LoRA consistently outperforms full fine-tuning, achieving 43.52 +/- 0.18 ROUGE-1 on Flan-T5-Large with only 0.6% trainable parameters compared to 40.67 +/- 0.21 for full fine-tuning.
• Score: 13.495539801703389
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning large language models for domain-specific tasks such as medical text summarization demands substantial computational resources. Parameter-efficient fine-tuning (PEFT) methods offer promising alternatives by updating only a small fraction of parameters. This paper compares three adaptation approaches-Low-Rank Adaptation (LoRA), Prompt Tuning, and Full Fine-Tuning-across the Flan-T5 model family on the PubMed medical summarization dataset. Through experiments with multiple random seeds, we demonstrate that LoRA consistently outperforms full fine-tuning, achieving 43.52 +/- 0.18 ROUGE-1 on Flan-T5-Large with only 0.6% trainable parameters compared to 40.67 +/- 0.21 for full fine-tuning. Sensitivity analyses examine the impact of LoRA rank and prompt token count. Our findings suggest the low-rank constraint provides beneficial regularization, challenging assumptions about the necessity of full parameter updates. Code is available at https://github.com/eracoding/llm-medical-summarization

Related papers

• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models [4.092357804635167]
  We propose a novel PEFT method that incorporates the TLoRA+ into the weight matrices of pre-trained models.<n>The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost.
  arXiv  Detail & Related papers  (2026-04-15T00:30:57Z)
• High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>We present textbfSMoA, a high-rank textbfStructured textbfMOdulation textbfAdapter that uses fewer trainable parameters while maintaining a higher rank.
  arXiv  Detail & Related papers  (2026-01-12T13:06:17Z)
• SALT: Parameter-Efficient Fine-Tuning via Singular Value Adaptation with Low-Rank Transformation [6.267971370719281]
  Large foundation models offer considerable flexibility, yet the cost of fine-tuning these models remains a significant barrier.<n>We propose SALT, a method that selectively adapts the most influential singular values using trainable scale and shift parameters.<n>This hybrid approach harnesses the advantages of both LoRA and SVD, enabling effective adaptation without relying on increasing model size or depth.
  arXiv  Detail & Related papers  (2025-03-20T11:42:41Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation [17.649599011681712]
  Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size.<n>This paper highlights the importance of effective parameterization in low-rank fine-tuning to reduce estimator variance and enhance the stability of final model outputs.<n>We propose MonteCLoRA, an efficient fine-tuning technique that employs Monte Carlo estimation to learn an unbiased posterior estimation of low-rank parameters with low expected variance.
  arXiv  Detail & Related papers  (2024-11-07T01:31:48Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models [3.7049613588433497]
  Low-Rank Adaptation (LoRA) significantly reduces the number of trainable parameters for fine-tuning. We extend the LoRA to multiple scales, dubbed as LoRA$2$.
  arXiv  Detail & Related papers  (2024-08-13T12:31:30Z)
• Scaling Exponents Across Parameterizations and Optimizers [94.54718325264218]
  We propose a new perspective on parameterization by investigating a key assumption in prior work. Our empirical investigation includes tens of thousands of models trained with all combinations of threes. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work.
  arXiv  Detail & Related papers  (2024-07-08T12:32:51Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability [73.34532767873785]
  We propose the concept of Information Density'' (ID) to indicate whether a matrix strongly belongs to certain feature spaces. We introduce the Dense Information Prompt (DIP) to enhance information density to improve generalization. DIP significantly reduces the number of tunable parameters and the requisite storage space, making it particularly advantageous in resource-constrained settings.
  arXiv  Detail & Related papers  (2023-12-17T20:42:43Z)
• When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications [57.342772288710044]
  We propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2023-10-21T17:18:09Z)
• Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling [5.601559340796398]
  We propose a framework that integrates LoRA and structured layer pruning. Our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase.
  arXiv  Detail & Related papers  (2023-05-15T00:21:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.