PEFT-MedAware: Large Language Model for Medical Awareness

• URL: http://arxiv.org/abs/2311.10697v1
• Date: Fri, 17 Nov 2023 18:32:17 GMT
• Title: PEFT-MedAware: Large Language Model for Medical Awareness
• Authors: Keivalya Pandya
• Abstract summary: We propose a specialized PEFT-MedAware model to enhance the Falcon-1b large language model on specialized MedQuAD data. The model was capable of outperforming other LLMs in medical question-answering tasks in specific domains. We propose further improvements through expanded datasets, larger models, and feedback mechanisms for sustained medical relevancy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chat models are capable of answering a wide range of questions, however, the accuracy of their responses is highly uncertain. In this research, we propose a specialized PEFT-MedAware model where we utilize parameter-efficient fine-tuning (PEFT) to enhance the Falcon-1b large language model on specialized MedQuAD data consisting of 16,407 medical QA pairs, leveraging only 0.44% of its trainable parameters to enhance computational efficiency. The paper adopts data preprocessing and PEFT to optimize model performance, complemented by a BitsAndBytesConfig for efficient transformer training. The resulting model was capable of outperforming other LLMs in medical question-answering tasks in specific domains with greater accuracy utilizing limited computational resources making it suitable for deployment in resource-constrained environments. We propose further improvements through expanded datasets, larger models, and feedback mechanisms for sustained medical relevancy. Our work highlights the efficiency gains and specialized capabilities of PEFT in medical AI, outpacing standard models in precision without extensive resource demands. The proposed model and data are released for research purposes only.

Related papers

• Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies [17.904502959675337]
  Large models require substantial computational resources and GPU memory to operate. Efficient Fine-Tuning (PEFT) offers a practical solution by efficiently adjusting the parameters of large pre-trained models to suit various downstream tasks. This review mainly introduces the preliminary knowledge of PEFT, the core ideas and principles of various PEFT algorithms, the applications of PEFT, and potential future research directions.
  arXiv  Detail & Related papers  (2024-10-24T13:58:59Z)
• Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification [16.070261684997362]
  Fine-tuning pre-trained models for various downstream tasks is a critical problem in the medical imaging domain. Large size of these models necessitates the use of parameter-efficient fine-tuning (PEFT) to reduce the communication burden in federated learning. In this work, we investigate various federated PEFT strategies for adapting a Vision Transformer (ViT) model for medical image classification.
  arXiv  Detail & Related papers  (2024-07-16T10:28:50Z)
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering [58.79671189792399]
  STLLaVA-Med is designed to train a policy model capable of auto-generating medical visual instruction data. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks.
  arXiv  Detail & Related papers  (2024-06-28T15:01:23Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks [2.834743715323873]
  We present an investigation into the suitability of different PEFT methods to clinical decision-making tasks. Our analysis shows that the performance of most PEFT approaches varies significantly from one task to another. The effectiveness of PEFT methods in the clinical domain is evident, particularly for specialised models which can operate on low-cost, in-house computing infrastructure.
  arXiv  Detail & Related papers  (2024-02-16T11:30:11Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization [100.90624220423634]
  We present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models. In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x.
  arXiv  Detail & Related papers  (2023-11-22T05:28:59Z)
• DVPT: Dynamic Visual Prompt Tuning of Large Pre-trained Models for Medical Image Analysis [30.608225734194416]
  We propose a dynamic visual prompt tuning method, named DVPT, for medical image analysis. It can extract knowledge beneficial to downstream tasks from large models with a few trainable parameters. It can save up to 60% labeled data and 99% storage cost of ViT-B/16.
  arXiv  Detail & Related papers  (2023-07-19T07:11:11Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• A Data and Compute Efficient Design for Limited-Resources Deep Learning [68.55415606184]
  equivariant neural networks have gained increased interest in the deep learning community. They have been successfully applied in the medical domain where symmetries in the data can be effectively exploited to build more accurate and robust models. Mobile, on-device implementations of deep learning solutions have been developed for medical applications. However, equivariant models are commonly implemented using large and computationally expensive architectures, not suitable to run on mobile devices. In this work, we design and test an equivariant version of MobileNetV2 and further optimize it with model quantization to enable more efficient inference.
  arXiv  Detail & Related papers  (2020-04-21T00:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.