Adapting LLMs for the Medical Domain in Portuguese: A Study on Fine-Tuning and Model Evaluation

• URL: http://arxiv.org/abs/2410.00163v1
• Date: Mon, 30 Sep 2024 19:10:03 GMT
• Title: Adapting LLMs for the Medical Domain in Portuguese: A Study on Fine-Tuning and Model Evaluation
• Authors: Pedro Henrique Paiola, Gabriel Lino Garcia, João Renato Ribeiro Manesco, Mateus Roder, Douglas Rodrigues, João Paulo Papa,
• Abstract summary: This study evaluates the performance of large language models (LLMs) as medical agents in Portuguese. The InternLM2 model, with initial training on medical data, presented the best overall performance. DrBode models, derived from ChatBode, exhibited a phenomenon of catastrophic forgetting of acquired medical knowledge.
• Score: 1.922611370494431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study evaluates the performance of large language models (LLMs) as medical agents in Portuguese, aiming to develop a reliable and relevant virtual assistant for healthcare professionals. The HealthCareMagic-100k-en and MedQuAD datasets, translated from English using GPT-3.5, were used to fine-tune the ChatBode-7B model using the PEFT-QLoRA method. The InternLM2 model, with initial training on medical data, presented the best overall performance, with high precision and adequacy in metrics such as accuracy, completeness and safety. However, DrBode models, derived from ChatBode, exhibited a phenomenon of catastrophic forgetting of acquired medical knowledge. Despite this, these models performed frequently or even better in aspects such as grammaticality and coherence. A significant challenge was low inter-rater agreement, highlighting the need for more robust assessment protocols. This work paves the way for future research, such as evaluating multilingual models specific to the medical field, improving the quality of training data, and developing more consistent evaluation methodologies for the medical field.

Related papers

• LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model [55.80651780294357]
  State-of-the-art medical multi-modal large language models (med-MLLM) leverage instruction-following data in pre-training. LoGra-Med is a new multi-graph alignment algorithm that enforces triplet correlations across image modalities, conversation-based descriptions, and extended captions. Our results show LoGra-Med matches LLAVA-Med performance on 600K image-text pairs for Medical VQA and significantly outperforms it when trained on 10% of the data.
  arXiv  Detail & Related papers  (2024-10-03T15:52:03Z)
• Enhancing Healthcare through Large Language Models: A Study on Medical Question Answering [13.237829215746443]
  Sentence-t5 + Mistral 7B model excels in understanding and generating precise medical answers. Our findings highlight the potential of integrating sophisticated LLMs in medical contexts.
  arXiv  Detail & Related papers  (2024-08-08T00:35:39Z)
• LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation [0.0]
  This study introduces a novel "LLMs-in-the-loop" approach to develop supervised neural machine translation models optimized for medical texts. Custom parallel corpora in six languages were compiled from scientific articles, synthetically generated clinical documents, and medical texts. Our MarianMT-based models outperform Google Translate, DeepL, and GPT-4-Turbo.
  arXiv  Detail & Related papers  (2024-07-16T19:32:23Z)
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering [58.79671189792399]
  STLLaVA-Med is designed to train a policy model capable of auto-generating medical visual instruction data. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks.
  arXiv  Detail & Related papers  (2024-06-28T15:01:23Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging [8.043625583479598]
  Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. We propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks.
  arXiv  Detail & Related papers  (2024-01-05T13:22:12Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)
• PMC-LLaMA: Towards Building Open-source Language Models for Medicine [62.39105735933138]
  Large Language Models (LLMs) have showcased remarkable capabilities in natural language understanding. LLMs struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. We describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA.
  arXiv  Detail & Related papers  (2023-04-27T18:29:05Z)
• MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain [45.96917694724562]
  medBERTde is a pre-trained German BERT model specifically designed for the German medical domain. The model has been trained on a large corpus of 4.7 Million German medical documents.
  arXiv  Detail & Related papers  (2023-03-14T18:58:08Z)
• Large Language Models Encode Clinical Knowledge [21.630872464930587]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning.
  arXiv  Detail & Related papers  (2022-12-26T14:28:24Z)
• Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks [62.9447303059342]
  We show the importance of this problem in medical community. We present a modification of Bidirectional Representations from Transformers (BERT) model for classification sequence. We use a large-scale Russian EHR dataset consisting of about 4 million unique patient visits.
  arXiv  Detail & Related papers  (2020-07-15T09:22:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.