JMedLoRA:Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning

• URL: http://arxiv.org/abs/2310.10083v2
• Date: Fri, 1 Dec 2023 00:29:37 GMT
• Title: JMedLoRA:Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning
• Authors: Issey Sukeda, Masahiro Suzuki, Hiroki Sakaji, Satoshi Kodera
• Abstract summary: We show the contribution of LoRA-based instruction-tuning to performance in Japanese medical question-answering tasks. Our results underscore the potential of adapting English-centric models for Japanese applications in domain adaptation.
• Score: 5.249703210938688
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the ongoing wave of impact driven by large language models (LLMs) like ChatGPT, the adaptation of LLMs to medical domain has emerged as a crucial research frontier. Since mainstream LLMs tend to be designed for general-purpose applications, constructing a medical LLM through domain adaptation is a huge challenge. While instruction-tuning is used to fine-tune some LLMs, its precise roles in domain adaptation remain unknown. Here we show the contribution of LoRA-based instruction-tuning to performance in Japanese medical question-answering tasks. In doing so, we employ a multifaceted evaluation for multiple-choice questions, including scoring based on "Exact match" and "Gestalt distance" in addition to the conventional accuracy. Our findings suggest that LoRA-based instruction-tuning can partially incorporate domain-specific knowledge into LLMs, with larger models demonstrating more pronounced effects. Furthermore, our results underscore the potential of adapting English-centric models for Japanese applications in domain adaptation, while also highlighting the persisting limitations of Japanese-centric models. This initiative represents a pioneering effort in enabling medical institutions to fine-tune and operate models without relying on external services.

Related papers

• Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources [0.0]
  We present a medical adaptation based on the recent 7B models, which enables the operation in low computational resources. We find that fine-tuning an English-centric base model on Japanese medical dataset improves the score in both language.
  arXiv  Detail & Related papers  (2024-09-18T08:07:37Z)
• 70B-parameter large language models in Japanese medical question-answering [0.0]
  We show that instruction tuning using Japanese medical question-answering dataset significantly improves the ability of Japanese LLMs to solve Japanese medical license exams. In particular, the Japanese-centric models exhibit a more significant leap in improvement through instruction tuning compared to their English-centric counterparts.
  arXiv  Detail & Related papers  (2024-06-21T06:04:10Z)
• BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models [56.89958793648104]
  Large Language Models (LLMs) are versatile and capable of addressing a diverse range of tasks. Previous approaches either conduct continuous pre-training with domain-specific data or employ retrieval augmentation to support general LLMs. We present a novel framework named BLADE, which enhances Black-box LArge language models with small Domain-spEcific models.
  arXiv  Detail & Related papers  (2024-03-27T08:57:21Z)
• Can Large Language Models abstract Medical Coded Language? [0.0]
  Large language models (LLMs) are aware of medical code and can accurately generate names from these codes. This study evaluates whether large language models (LLMs) are aware of medical code and can accurately generate names from these codes.
  arXiv  Detail & Related papers  (2024-03-16T06:18:15Z)
• PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs [49.32067576992511]
  Large language models often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. We propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA) Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks.
  arXiv  Detail & Related papers  (2024-02-20T09:02:55Z)
• Large Language Model Distilling Medication Recommendation Model [61.89754499292561]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs) Our research aims to transform existing medication recommendation methodologies using LLMs. To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge [28.409333447902693]
  Large language models (LLMs) achieve remarkable performance in comprehensive and generative ability. In this work, we focus on the particular disease of Epilepsy with Japanese language and introduce a customized LLM termed as EpilepsyLLM. The datasets contain knowledge of basic information about disease, common treatment methods and drugs, and important notes in life and work.
  arXiv  Detail & Related papers  (2024-01-11T13:39:00Z)
• Low-Rank Adaptation for Multilingual Summarization: An Empirical Study [60.541168233698194]
  We investigate the potential of. Efficient Fine-Tuning, focusing on Low-Rank Adaptation (LoRA) in the domain of multilingual summarization. We conduct an extensive study across different data availability scenarios, including high- and low-data settings, and cross-lingual transfer. Our findings reveal that LoRA is competitive with full fine-tuning when trained with high quantities of data, and excels in low-data scenarios and cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-11-14T22:32:39Z)
• ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences [51.66185471742271]
  We propose ChiMed-GPT, a benchmark LLM designed explicitly for Chinese medical domain. ChiMed-GPT undergoes a comprehensive training regime with pre-training, SFT, and RLHF. We analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients.
  arXiv  Detail & Related papers  (2023-11-10T12:25:32Z)
• A Survey of Large Language Models in Medicine: Progress, Application, and Challenge [85.09998659355038]
  Large language models (LLMs) have received substantial attention due to their capabilities for understanding and generating human language. This review aims to provide a detailed overview of the development and deployment of LLMs in medicine.
  arXiv  Detail & Related papers  (2023-11-09T02:55:58Z)
• Improving Small Language Models on PubMedQA via Generative Data Augmentation [4.96649519549027]
  Large Language Models (LLMs) have made remarkable advancements in the field of natural language processing. Small Language Models (SLMs) are known for their efficiency, but they often struggle with limited capacity and training data. We introduce a novel method aimed at improving SLMs in the medical domain using LLM-based generative data augmentation.
  arXiv  Detail & Related papers  (2023-05-12T23:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.