Technical Report: Small Language Model for Japanese Clinical and Medicine

• URL: http://arxiv.org/abs/2412.16423v1
• Date: Sat, 21 Dec 2024 01:12:48 GMT
• Title: Technical Report: Small Language Model for Japanese Clinical and Medicine
• Authors: Shogo Watanabe,
• Abstract summary: This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. In comparison to other large language models, a fine-tuning NCVC-slm-1 demonstrated the highest scores on 6 tasks of total 8 on JMED-LLM.
• Score: 0.0
• License:
• Abstract: This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. This 1B parameters model was trained using Japanese text classified to be of high-quality. Moreover, NCVC-slm-1 was augmented with respect to clinical and medicine content that includes the variety of diseases, drugs, and examinations. Using a carefully designed pre-processing, a specialized morphological analyzer and tokenizer, this small and light-weight model performed not only to generate text but also indicated the feasibility of understanding clinical and medicine text. In comparison to other large language models, a fine-tuning NCVC-slm-1 demonstrated the highest scores on 6 tasks of total 8 on JMED-LLM. According to this result, SLM indicated the feasibility of performing several downstream tasks in the field of clinical and medicine. Hopefully, NCVC-slm-1 will be contributed to develop and accelerate the field of clinical and medicine for a bright future.

Related papers

• CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios [50.032101237019205]
  CliMedBench is a comprehensive benchmark with 14 expert-guided core clinical scenarios. The reliability of this benchmark has been confirmed in several ways.
  arXiv  Detail & Related papers  (2024-10-04T15:15:36Z)
• Towards Evaluating and Building Versatile Large Language Models for Medicine [57.49547766838095]
  We present MedS-Bench, a benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks.
  arXiv  Detail & Related papers  (2024-08-22T17:01:34Z)
• ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes [6.921652448124103]
  We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes. With 130 million and 2.8 billion parameters, ClinicalMamba demonstrates a superior performance in modeling clinical language across extended text lengths.
  arXiv  Detail & Related papers  (2024-03-09T04:58:25Z)
• Large Language Model Distilling Medication Recommendation Model [58.94186280631342]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs) Our research aims to transform existing medication recommendation methodologies using LLMs. To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Models [7.698164945017469]
  textbfMedTem project uses several advanced learning structures including BiLSTM-CRF and CNN-BiLSTM. CNN-BiLSTM slightly wins the BiLSTM-CRF model on the i2b2-2009 clinical NER task yielding 75.67, 77.83, and 78.17 for precision, recall, and F1 scores. BERT-CNN model also produced reasonable evaluation scores 64.48, 67.17, and 65.03 for P/R/F1 using Macro Avg.
  arXiv  Detail & Related papers  (2023-10-03T17:37:22Z)
• A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks [7.542019351929903]
  We evaluate four state-of-the-art instruction-tuned large language models (LLMs) On a set of 13 real-world clinical and biomedical natural language processing (NLP) tasks in English.
  arXiv  Detail & Related papers  (2023-07-22T15:58:17Z)
• Do We Still Need Clinical Language Models? [15.023633270864675]
  We show that relatively small specialized clinical models substantially outperform all in-context learning approaches. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.
  arXiv  Detail & Related papers  (2023-02-16T05:08:34Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records [22.652798872046283]
  There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records ( EHRs) There are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters. It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs.
  arXiv  Detail & Related papers  (2022-02-02T14:28:51Z)
• CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark [51.38557174322772]
  We present the first Chinese Biomedical Language Understanding Evaluation benchmark. It is a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification. We report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.
  arXiv  Detail & Related papers  (2021-06-15T12:25:30Z)
• Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation [48.87254340298189]
  We construct a new dataset named MedLane to support the development and evaluation of automated clinical language simplification approaches. We propose a new model called DECLARE that follows the human annotation procedure and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-12-04T06:09:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.