How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling

• URL: http://arxiv.org/abs/2211.07713v1
• Date: Tue, 25 Oct 2022 09:21:28 GMT
• Title: How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling
• Authors: Samuel Cahyawijaya, Bryan Wilie, Holy Lovenia, Huan Zhong, MingQian Zhong, Yuk-Yu Nancy Ip, Pascale Fung
• Abstract summary: Large pre-trained language models (LMs) have been widely adopted in biomedical and clinical domains. This work explores long-range adaptation from such LMs with Longformer, allowing the LMs to capture longer clinical notes context. We conduct experiments on three n2c2 challenges datasets and a longitudinal clinical dataset from Hong Kong Hospital Authority electronic health record system.
• Score: 37.247872987053654
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large pre-trained language models (LMs) have been widely adopted in biomedical and clinical domains, introducing many powerful LMs such as bio-lm and BioELECTRA. However, the applicability of these methods to real clinical use cases is hindered, due to the limitation of pre-trained LMs in processing long textual data with thousands of words, which is a common length for a clinical note. In this work, we explore long-range adaptation from such LMs with Longformer, allowing the LMs to capture longer clinical notes context. We conduct experiments on three n2c2 challenges datasets and a longitudinal clinical dataset from Hong Kong Hospital Authority electronic health record (EHR) system to show the effectiveness and generalizability of this concept, achieving 10\% F1-score improvement. Based on our experiments, we conclude that capturing a longer clinical note interval is beneficial to the model performance, but there are different cut-off intervals to achieve the optimal performance for different target variables. Our code is available at https://github.com/HLTCHKUST/long-biomedical-model.

Related papers

• LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models [61.12177317970258]
  LongSkywork is a long-context Large Language Model capable of processing up to 200,000 tokens. We develop two novel methods for creating synthetic data. LongSkywork achieves outstanding performance on a variety of long-context benchmarks.
  arXiv  Detail & Related papers  (2024-06-02T03:34:41Z)
• TrialDura: Hierarchical Attention Transformer for Interpretable Clinical Trial Duration Prediction [19.084936647082632]
  We propose TrialDura, a machine learning-based method that estimates the duration of clinical trials using multimodal data. We encode them into Bio-BERT embeddings specifically tuned for biomedical contexts to provide a deeper and more relevant semantic understanding. Our proposed model demonstrated superior performance with a mean absolute error (MAE) of 1.04 years and a root mean square error (RMSE) of 1.39 years compared to the other models.
  arXiv  Detail & Related papers  (2024-04-20T02:12:59Z)
• Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study [4.042419725040222]
  Pretrained language models based on BERT have been introduced for the French biomedical domain. These models are constrained by a limited input sequence length of 512 tokens, which poses challenges when applied to clinical notes. We present a comparative study of three adaptation strategies for long-sequence models, leveraging the Longformer architecture.
  arXiv  Detail & Related papers  (2024-02-26T16:05:33Z)
• LongBoX: Evaluating Transformers on Long-Sequence Clinical Tasks [44.89857441408805]
  LongBoX is a collection of seven medical datasets in text-to-text format. Preliminary experiments reveal that both medical LLMs and strong general domain LLMs struggle on this benchmark. We evaluate two techniques designed for long-sequence handling: (i) local-global attention, and (ii) Fusion-in-Decoder (FiD)
  arXiv  Detail & Related papers  (2023-11-16T04:57:49Z)
• Making the Most Out of the Limited Context Length: Predictive Power Varies with Clinical Note Type and Note Section [70.37720062263176]
  We propose a framework to analyze the sections with high predictive power. Using MIMIC-III, we show that: 1) predictive power distribution is different between nursing notes and discharge notes and 2) combining different types of notes could improve performance when the context length is large.
  arXiv  Detail & Related papers  (2023-07-13T20:04:05Z)
• Time Associated Meta Learning for Clinical Prediction [78.99422473394029]
  We propose a novel time associated meta learning (TAML) method to make effective predictions at multiple future time points. To address the sparsity problem after task splitting, TAML employs a temporal information sharing strategy to augment the number of positive samples. We demonstrate the effectiveness of TAML on multiple clinical datasets, where it consistently outperforms a range of strong baselines.
  arXiv  Detail & Related papers  (2023-03-05T03:54:54Z)
• A Comparative Study of Pretrained Language Models for Long Clinical Text [4.196346055173027]
  We introduce two domain enriched language models, Clinical-Longformer and Clinical-BigBird, which are pre-trained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks.
  arXiv  Detail & Related papers  (2023-01-27T16:50:29Z)
• Cross-Lingual Knowledge Transfer for Clinical Phenotyping [55.92262310716537]
  We investigate cross-lingual knowledge transfer strategies to execute this task for clinics that do not use the English language. We evaluate these strategies for a Greek and a Spanish clinic leveraging clinical notes from different clinical domains. Our results show that using multilingual data overall improves clinical phenotyping models and can compensate for data sparseness.
  arXiv  Detail & Related papers  (2022-08-03T08:33:21Z)
• Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences [4.196346055173027]
  Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. We introduce two domain enriched language models, namely Clinical-Longformer and Clinical-BigBird, which are pre-trained from large-scale clinical corpora.
  arXiv  Detail & Related papers  (2022-01-27T22:51:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.