Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record

• URL: http://arxiv.org/abs/2303.14920v1
• Date: Mon, 27 Mar 2023 05:34:19 GMT
• Title: Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record
• Authors: Christopher McMaster, David FL Liew, Douglas EV Pires
• Abstract summary: We pretrain a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. We compare this model's performance with a DeBERTa model pre-trained on clinical texts from our institutional EHR (DeBERTa) and an XGBoost model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose an approach for adapting the DeBERTa model for electronic health record (EHR) tasks using domain adaptation. We pretrain a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. We compare this model's performance with a DeBERTa model pre-trained on clinical texts from our institutional EHR (MeDeBERTa) and an XGBoost model. We evaluate performance on three benchmark tasks for emergency department outcomes using the MIMIC-IV-ED dataset. We preprocess the data to convert it into text format and generate four versions of the original datasets to compare data processing and data inclusion. The results show that our proposed approach outperforms the alternative models on two of three tasks (p<0.001) and matches performance on the third task, with the use of descriptive columns improving performance over the original column names.

Related papers

• A Three-Phases SFT Hybrid Model Integrated Strong Prior Module and Data Overlap Estimation in the Eduation Context [0.0]
  We propose an end-to-end prior-based three-phases supervised fine-tuned model. Our model realizes the structural disassembly and incremental guided output of educational knowledge. Our model also achieves state-of-the-art in code abilities compared to open-source models.
  arXiv  Detail & Related papers  (2024-03-13T05:38:39Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models [6.506937003687058]
  We publish a new dataset, EHRSHOT, which contains deidentified structured data from the electronic health records (EHRs) of 6,739 patients from Stanford Medicine. Second, we publish the weights of CLMBR-T-base, a 141M parameter clinical foundation model pretrained on the structured EHR data of 2.57M patients. Third, we define 15 few-shot clinical prediction tasks, enabling evaluation of foundation models on benefits such as sample efficiency and task adaptation.
  arXiv  Detail & Related papers  (2023-07-05T05:24:59Z)
• Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models [59.89454513692417]
  Tabular data is often hidden in text, particularly in medical diagnostic reports. We propose a novel, simple, and effective methodology for extracting structured tabular data from textual medical reports, called TEMED-LLM. We demonstrate that our approach significantly outperforms state-of-the-art text classification models in medical diagnostics.
  arXiv  Detail & Related papers  (2023-06-08T09:12:28Z)
• Stacking Ensemble Learning in Deep Domain Adaptation for Ophthalmic Image Classification [61.656149405657246]
  Domain adaptation is effective in image classification tasks where obtaining sufficient label data is challenging. We propose a novel method, named SELDA, for stacking ensemble learning via extending three domain adaptation methods. The experimental results using Age-Related Eye Disease Study (AREDS) benchmark ophthalmic dataset demonstrate the effectiveness of the proposed model.
  arXiv  Detail & Related papers  (2022-09-27T14:19:00Z)
• Negation detection in Dutch clinical texts: an evaluation of rule-based and machine learning methods [0.21079694661943607]
  We compare three methods for negation detection in Dutch clinical notes. We found that both the biLSTM and RoBERTa models consistently outperform the rule-based model in terms of F1 score, precision and recall.
  arXiv  Detail & Related papers  (2022-09-01T14:00:13Z)
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records [57.75125067744978]
  We propose a data augmentation method to facilitate domain adaptation. adversarially generated samples are used during domain adaptation. Results confirm the effectiveness of our method and the generality on different tasks.
  arXiv  Detail & Related papers  (2021-01-13T03:20:20Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)
• Machine learning with incomplete datasets using multi-objective optimization models [1.933681537640272]
  We propose an online approach to handle missing values while a classification model is learnt. We develop a multi-objective optimization model with two objective functions for imputation and model selection. We use an evolutionary algorithm based on NSGA II to find the optimal solutions.
  arXiv  Detail & Related papers  (2020-12-04T03:44:33Z)
• An Ensemble Approach for Automatic Structuring of Radiology Reports [6.186392239590685]
  We present an ensemble method that consolidates the predictions of three models, capturing various attributes of textual information for automatic labeling of sentences. Our proposed approach significantly outperforms other approaches by achieving 97.1% accuracy.
  arXiv  Detail & Related papers  (2020-10-05T18:11:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.