Multimodal Medical Code Tokenizer

• URL: http://arxiv.org/abs/2502.04397v2
• Date: Wed, 12 Feb 2025 22:26:50 GMT
• Title: Multimodal Medical Code Tokenizer
• Authors: Xiaorui Su, Shvat Messica, Yepeng Huang, Ruth Johnson, Lukas Fesser, Shanghua Gao, Faryad Sahneh, Marinka Zitnik,
• Abstract summary: Existing tokenizers treat medical codes from EHRs as isolated textual tokens.<n>Medical vocabularies contain more than 600,000 codes with critical information for clinical reasoning.<n>We introduce MedTok, a multimodal medical code tokenizer that uses the text descriptions and relational context of codes.
• Score: 15.816571598837823
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models trained on patient electronic health records (EHRs) require tokenizing medical data into sequences of discrete vocabulary items. Existing tokenizers treat medical codes from EHRs as isolated textual tokens. However, each medical code is defined by its textual description, its position in ontological hierarchies, and its relationships to other codes, such as disease co-occurrences and drug-treatment associations. Medical vocabularies contain more than 600,000 codes with critical information for clinical reasoning. We introduce MedTok, a multimodal medical code tokenizer that uses the text descriptions and relational context of codes. MedTok processes text using a language model encoder and encodes the relational structure with a graph encoder. It then quantizes both modalities into a unified token space, preserving modality-specific and cross-modality information. We integrate MedTok into five EHR models and evaluate it on operational and clinical tasks across in-patient and out-patient datasets, including outcome prediction, diagnosis classification, drug recommendation, and risk stratification. Swapping standard EHR tokenizers with MedTok improves AUPRC across all EHR models, by 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.30% on EHRShot, with the largest gains in drug recommendation. Beyond EHR modeling, we demonstrate using MedTok tokenizer with medical QA systems. Our results demonstrate the potential of MedTok as a unified tokenizer for medical codes, improving tokenization for medical foundation models.

Related papers

• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• Natural Language-Assisted Multi-modal Medication Recommendation [97.07805345563348]
  We introduce the Natural Language-Assisted Multi-modal Medication Recommendation(NLA-MMR)<n>The NLA-MMR is a multi-modal alignment framework designed to learn knowledge from the patient view and medication view jointly.<n>In this vein, we employ pretrained language models(PLMs) to extract in-domain knowledge regarding patients and medications.
  arXiv  Detail & Related papers  (2025-01-13T09:51:50Z)
• MedCodER: A Generative AI Assistant for Medical Coding [3.7153274758003967]
  We introduce MedCodER, a Generative AI framework for automatic medical coding. MedCodER achieves a micro-F1 score of 0.60 on International Classification of Diseases (ICD) code prediction. We present a new dataset containing medical records annotated with disease diagnoses, ICD codes, and supporting evidence texts.
  arXiv  Detail & Related papers  (2024-09-18T19:36:33Z)
• Towards Evaluating and Building Versatile Large Language Models for Medicine [57.49547766838095]
  We present MedS-Bench, a benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks.
  arXiv  Detail & Related papers  (2024-08-22T17:01:34Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding [72.18719355481052]
  We introduce a novel task called Medical Report Grounding (MRG)<n>MRG aims to directly identify diagnostic phrases and their corresponding grounding boxes from medical reports in an end-to-end manner.<n>We propose uMedGround, a robust and reliable framework that leverages a multimodal large language model to predict diagnostic phrases.
  arXiv  Detail & Related papers  (2024-04-10T07:41:35Z)
• Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines [0.42641920138420947]
  We present our Read, Attend, and Code (RAC) model for learning the medical code assignment mappings. RAC establishes a new state of the art (SOTA) considerably outperforming the current best Macro-F1 by 18.7%. This new milestone marks a meaningful step toward fully autonomous medical coding (AMC) in machines.
  arXiv  Detail & Related papers  (2021-07-10T06:01:58Z)
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study [2.871614744079523]
  It is not clear if pretrained models are useful for medical code prediction without further architecture engineering. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes.
  arXiv  Detail & Related papers  (2021-03-11T07:23:45Z)
• A Meta-embedding-based Ensemble Approach for ICD Coding Prediction [64.42386426730695]
  International Classification of Diseases (ICD) are the de facto codes used globally for clinical coding. These codes enable healthcare providers to claim reimbursement and facilitate efficient storage and retrieval of diagnostic information. Our proposed approach enhances the performance of neural models by effectively training word vectors using routine medical data as well as external knowledge from scientific articles.
  arXiv  Detail & Related papers  (2021-02-26T17:49:58Z)
• An Explainable CNN Approach for Medical Codes Prediction from Clinical Text [1.7746314978241657]
  We develop CNN-based methods for automatic ICD coding based on clinical text from intensive care unit (ICU) stays. We come up with the Shallow and Wide Attention convolutional Mechanism (SWAM), which allows our model to learn local and low-level features for each label.
  arXiv  Detail & Related papers  (2021-01-14T02:05:34Z)
• Medical Code Assignment with Gated Convolution and Note-Code Interaction [39.079615516043674]
  We propose a novel method, gated convolutional neural networks, and a note-code interaction (GatedCNN-NCI) for automatic medical code assignment. With a novel note-code interaction design and a graph message passing mechanism, we explicitly capture the underlying dependency between notes and codes. Our proposed model outperforms state-of-the-art models in most cases, and our model size is on par with light-weighted baselines.
  arXiv  Detail & Related papers  (2020-10-14T11:37:24Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)
• DeepEnroll: Patient-Trial Matching with Deep Embedding and Entailment Prediction [67.91606509226132]
  Clinical trials are essential for drug development but often suffer from expensive, inaccurate and insufficient patient recruitment. DeepEnroll is a cross-modal inference learning model to jointly encode enrollment criteria (tabular data) into a shared latent space for matching inference.
  arXiv  Detail & Related papers  (2020-01-22T17:51:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.