NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text

• URL: http://arxiv.org/abs/2412.11477v1
• Date: Mon, 16 Dec 2024 06:44:39 GMT
• Title: NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text
• Authors: Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae,
• Abstract summary: We develop an approach based on i) models for ICD-10 diagnostic code sequences using a large real-world data set, ii) large language models for medical notes, andiii) contrastive pre-training to build an integrated model of both ICD-10 diagnostic codes and corresponding medical text. We demonstrate that a contrastive approach for pre-training improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks.
• Score: 0.0
• License:
• Abstract: Accurate diagnostic coding of medical notes is crucial for enhancing patient care, medical research, and error-free billing in healthcare organizations. Manual coding is a time-consuming task for providers, and diagnostic codes often exhibit low sensitivity and specificity, whereas the free text in medical notes can be a more precise description of a patients status. Thus, accurate automated diagnostic coding of medical notes has become critical for a learning healthcare system. Recent developments in long-document transformer architectures have enabled attention-based deep-learning models to adjudicate medical notes. In addition, contrastive loss functions have been used to jointly pre-train large language and image models with noisy labels. To further improve the automated adjudication of medical notes, we developed an approach based on i) models for ICD-10 diagnostic code sequences using a large real-world data set, ii) large language models for medical notes, and iii) contrastive pre-training to build an integrated model of both ICD-10 diagnostic codes and corresponding medical text. We demonstrate that a contrastive approach for pre-training improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks.

Related papers

• MedCodER: A Generative AI Assistant for Medical Coding [3.7153274758003967]
  We introduce MedCodER, a Generative AI framework for automatic medical coding. MedCodER achieves a micro-F1 score of 0.60 on International Classification of Diseases (ICD) code prediction. We present a new dataset containing medical records annotated with disease diagnoses, ICD codes, and supporting evidence texts.
  arXiv  Detail & Related papers  (2024-09-18T19:36:33Z)
• A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning [11.817595076396925]
  Diagnostic Captioning (DC) automatically generates a diagnostic text from one or more medical images of a patient. We propose a new data-driven guided decoding method that incorporates medical information into the beam search of the diagnostic text generation process. We evaluate the proposed method on two medical datasets using four DC systems that range from generic image-to-text systems with CNN encoders to pre-trained Large Language Models.
  arXiv  Detail & Related papers  (2024-06-20T10:08:17Z)
• CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning [56.782963838838036]
  We propose a novel approach, a contextualized and flexible framework, to enhance the learning of ICD code representations. Our approach employs a dependent learning paradigm that considers the context of clinical notes in modeling all possible code relations.
  arXiv  Detail & Related papers  (2024-02-24T03:25:28Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study [60.56194508762205]
  We reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. We present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models.
  arXiv  Detail & Related papers  (2023-04-21T11:54:44Z)
• MIMO: Mutual Integration of Patient Journey and Medical Ontology for Healthcare Representation Learning [49.57261599776167]
  We propose an end-to-end robust Transformer-based solution, Mutual Integration of patient journey and Medical Ontology (MIMO) for healthcare representation learning and predictive analytics.
  arXiv  Detail & Related papers  (2021-07-20T07:04:52Z)
• Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines [0.42641920138420947]
  We present our Read, Attend, and Code (RAC) model for learning the medical code assignment mappings. RAC establishes a new state of the art (SOTA) considerably outperforming the current best Macro-F1 by 18.7%. This new milestone marks a meaningful step toward fully autonomous medical coding (AMC) in machines.
  arXiv  Detail & Related papers  (2021-07-10T06:01:58Z)
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study [2.871614744079523]
  It is not clear if pretrained models are useful for medical code prediction without further architecture engineering. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes.
  arXiv  Detail & Related papers  (2021-03-11T07:23:45Z)
• Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation [4.4036730220831535]
  Recent studies on deep learning for automated medical coding achieved promising performances. We propose a Hierarchical Label-wise Attention Network (HLAN), which aimed to interpret the model by quantifying importance (as attention weights) of words and sentences related to each of the labels. Secondly, we propose to enhance the major deep learning models with a label embedding (LE) initialisation approach, which learns a dense, continuous vector representation and then injects the representation into the final layers and the label-wise attention layers in the models.
  arXiv  Detail & Related papers  (2020-10-29T16:21:26Z)
• MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation [86.38736781043109]
  We build and release a large-scale high-quality Medical Dialogue dataset related to 12 types of common Gastrointestinal diseases named MedDG. We propose two kinds of medical dialogue tasks based on MedDG dataset. One is the next entity prediction and the other is the doctor response generation. Experimental results show that the pre-train language models and other baselines struggle on both tasks with poor performance in our dataset.
  arXiv  Detail & Related papers  (2020-10-15T03:34:33Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.