Transfer Learning with Clinical Concept Embeddings from Large Language Models

• URL: http://arxiv.org/abs/2409.13893v1
• Date: Fri, 20 Sep 2024 20:50:55 GMT
• Title: Transfer Learning with Clinical Concept Embeddings from Large Language Models
• Authors: Yuhe Gao, Runxue Bao, Yuelyu Ji, Yiming Sun, Chenxi Song, Jeffrey P. Ferraro, Ye Ye,
• Abstract summary: Large Language Models (LLMs) show significant potential of capturing the semantic meaning of clinical concepts. This study analyzed electronic health records from two large healthcare systems to assess the impact of semantic embeddings on local, shared, and transfer learning models.
• Score: 4.838020198075334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge sharing is crucial in healthcare, especially when leveraging data from multiple clinical sites to address data scarcity, reduce costs, and enable timely interventions. Transfer learning can facilitate cross-site knowledge transfer, but a major challenge is heterogeneity in clinical concepts across different sites. Large Language Models (LLMs) show significant potential of capturing the semantic meaning of clinical concepts and reducing heterogeneity. This study analyzed electronic health records from two large healthcare systems to assess the impact of semantic embeddings from LLMs on local, shared, and transfer learning models. Results indicate that domain-specific LLMs, such as Med-BERT, consistently outperform in local and direct transfer scenarios, while generic models like OpenAI embeddings require fine-tuning for optimal performance. However, excessive tuning of models with biomedical embeddings may reduce effectiveness, emphasizing the need for balance. This study highlights the importance of domain-specific embeddings and careful model tuning for effective knowledge transfer in healthcare.

Related papers

• Domain-invariant Clinical Representation Learning by Bridging Data Distribution Shift across EMR Datasets [16.317118701435742]
  An effective prognostic model is expected to assist doctors in making right diagnosis and designing personalized treatment plan. In the early stage of a disease, limited data collection and clinical experiences, plus the concern out of privacy and ethics, may result in restricted data availability for reference. This article introduces a domain-invariant representation learning method to build a transition model from source dataset to target dataset.
  arXiv  Detail & Related papers  (2023-10-11T18:32:21Z)
• Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning [10.212217551908525]
  We study the effectiveness of data and model scaling and cross-dataset knowledge transfer in a real-world pathology classification task. We identify the challenges of possible negative transfer and emphasize the significance of some key components. Our findings indicate a small and generic model (e.g. ShallowNet) performs well on a single dataset, however, a larger model (e.g. TCN) performs better on transfer and learning from a larger and diverse dataset.
  arXiv  Detail & Related papers  (2023-09-19T20:09:15Z)
• Improving Multiple Sclerosis Lesion Segmentation Across Clinical Sites: A Federated Learning Approach with Noise-Resilient Training [75.40980802817349]
  Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. We introduce a Decoupled Hard Label Correction (DHLC) strategy that considers the imbalanced distribution and fuzzy boundaries of MS lesions. We also introduce a Centrally Enhanced Label Correction (CELC) strategy, which leverages the aggregated central model as a correction teacher for all sites.
  arXiv  Detail & Related papers  (2023-08-31T00:36:10Z)
• Hierarchical Pretraining for Biomedical Term Embeddings [4.69793648771741]
  We propose HiPrBERT, a novel biomedical term representation model trained on hierarchical data. We show that HiPrBERT effectively learns the pair-wise distance from hierarchical information, resulting in a substantially more informative embeddings for further biomedical applications.
  arXiv  Detail & Related papers  (2023-07-01T08:16:00Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Maximizing Model Generalization for Machine Condition Monitoring with Self-Supervised Learning and Federated Learning [4.214064911004321]
  Deep Learning can diagnose faults and assess machine health from raw condition monitoring data without manually designed statistical features. Traditional supervised learning may struggle to learn compact, discriminative representations that generalize to unseen target domains. This study proposes focusing on maximizing the feature generality on the source domain and applying TL via weight transfer to copy the model to the target domain.
  arXiv  Detail & Related papers  (2023-04-27T17:57:54Z)
• LifeLonger: A Benchmark for Continual Disease Classification [59.13735398630546]
  We introduce LifeLonger, a benchmark for continual disease classification on the MedMNIST collection. Task and class incremental learning of diseases address the issue of classifying new samples without re-training the models from scratch. Cross-domain incremental learning addresses the issue of dealing with datasets originating from different institutions while retaining the previously obtained knowledge.
  arXiv  Detail & Related papers  (2022-04-12T12:25:05Z)
• When Accuracy Meets Privacy: Two-Stage Federated Transfer Learning Framework in Classification of Medical Images on Limited Data: A COVID-19 Case Study [77.34726150561087]
  COVID-19 pandemic has spread rapidly and caused a shortage of global medical resources. CNN has been widely utilized and verified in analyzing medical images.
  arXiv  Detail & Related papers  (2022-03-24T02:09:41Z)
• 2021 BEETL Competition: Advancing Transfer Learning for Subject Independence & Heterogenous EEG Data Sets [89.84774119537087]
  We design two transfer learning challenges around diagnostics and Brain-Computer-Interfacing (BCI) Task 1 is centred on medical diagnostics, addressing automatic sleep stage annotation across subjects. Task 2 is centred on Brain-Computer Interfacing (BCI), addressing motor imagery decoding across both subjects and data sets.
  arXiv  Detail & Related papers  (2022-02-14T12:12:20Z)
• Democratizing Artificial Intelligence in Healthcare: A Study of Model Development Across Two Institutions Incorporating Transfer Learning [8.043077408518826]
  Transfer learning (TL) allows a fully trained model from one institution to be fine-tuned by another institution using a much small local dataset. This report describes the challenges, methodology, and benefits of TL within the context of developing an AI model for a basic use-case.
  arXiv  Detail & Related papers  (2020-09-25T21:12:50Z)
• Uncovering the structure of clinical EEG signals with self-supervised learning [64.4754948595556]
  Supervised learning paradigms are often limited by the amount of labeled data that is available. This phenomenon is particularly problematic in clinically-relevant data, such as electroencephalography (EEG) By extracting information from unlabeled data, it might be possible to reach competitive performance with deep neural networks.
  arXiv  Detail & Related papers  (2020-07-31T14:34:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.