Tokenization Tradeoffs in Structured EHR Foundation Models

• URL: http://arxiv.org/abs/2603.15644v1
• Date: Tue, 03 Mar 2026 22:48:39 GMT
• Title: Tokenization Tradeoffs in Structured EHR Foundation Models
• Abstract summary: Tokenization determines what information is preserved, how efficiently it is encoded, and which relationships must be learned versus precomputed.<n>Here, we pretrained a transformer on pediatric EHR data under a factorial design, varying tokenization along event encoding, time encoding, and workflow annotation.<n>Joint event encoding and positional time encoding outperformed their alternatives while requiring 39.5% and 9.6% fewer pretraining floating-point operations, respectively.
• Score: 0.23453441553817042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models for structured electronic health records (EHRs) are pretrained on longitudinal sequences of timestamped clinical events to learn adaptable patient representations. Tokenization -- how these timelines are converted into discrete model inputs -- determines what information is preserved, how efficiently it is encoded, and which relationships must be learned versus precomputed. Yet the impact of tokenization design choices on downstream performance and computational efficiency remains largely unexplored. Here, we pretrained a transformer on pediatric EHR data under a factorial design, varying tokenization along event encoding, time encoding, and workflow annotation. We evaluated area-under-the-receiver-operating-characteristic curve across 74 clinical prediction tasks. Joint event encoding and positional time encoding outperformed their alternatives (73/74 and 71/74 tasks) while requiring 39.5% and 9.6% fewer pretraining floating-point operations, respectively. Targeted ablations traced the joint encoding advantage to local binding efficiency, that is, code-attribute pairs are combined into single tokens, rather than split across tokens that the model must learn to associate during pretraining. External evaluation on an adult intensive care unit cohort demonstrated that this advantage generalizes despite substantial vocabulary mismatch, while temporal and workflow effects remain institution-specific. These results establish tokenization as a tractable lever for improving both the performance and efficiency of EHR foundation models.

Related papers

• PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models [0.27998963147546146]
  We introduce PORTER, a structured EHR foundation model that decouples event representation from a fixed vocabulary.<n>Porter represents events through their descriptions using a frozen text encoder, integrates numeric values through a dedicated pathway, and learns clinical dynamics over patient timelines.<n>Porter achieved higher AUROC than a task-specific text serialization, at 329-fold lower amortized compute.
  arXiv  Detail & Related papers  (2026-06-23T03:33:25Z)
• Using Explainability as a Training-Time Reliability Signal for Efficient ECG Classification [14.279568423870572]
  We introduce ERTS, an explainability-based reliability training signal for efficient ECG classification.<n>ERTS uses explanation quality during training to distinguish between informative and unreliable uncertainty.<n>We evaluate ERTS across three ECG datasets and multiple backbone architectures, showing consistent improvements in macro-F1 alongside reduced effective training cost.
  arXiv  Detail & Related papers  (2026-06-10T15:55:00Z)
• SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization [55.63565289584336]
  SimReg is an embedding similarity regularization loss that explicitly encourages token representations with the same ground-truth label within each sequence to be more similar.<n>Our analysis reveals that this mechanism introduces gains by enlarging multi-classification margins, thereby enabling more efficient classification.
  arXiv  Detail & Related papers  (2026-05-09T08:59:13Z)
• Self-Supervised Learning via Flow-Guided Neural Operator on Time-Series Data [57.85958428020496]
  Flow-Guided Neural Operator (FGNO) is a novel framework combining operator learning with flow matching for SSL training.<n>FGNO learns mappings in functional spaces by using Short-Time Fourier Transform to unify different time resolutions.<n>Unlike prior generative SSL methods that use noisy inputs during inference, we propose using clean inputs for representation extraction while learning representations with noise.
  arXiv  Detail & Related papers  (2026-02-12T18:54:57Z)
• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• Rethinking Tokenization for Clinical Time Series: When Less is More [0.20388938295521575]
  We present a systematic evaluation of tokenization approaches for clinical time series modeling.<n>We show that explicit time encodings provide no consistent statistically significant benefit for the evaluated downstream tasks.<n>Value features show task-dependent importance, affecting mortality prediction but not readmission, suggesting code sequences alone can carry sufficient predictive signal.
  arXiv  Detail & Related papers  (2025-12-04T19:39:10Z)
• PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units [0.3277163122167433]
  We present PULSE-ICU, a self-supervised foundation model that learns event-level ICU representations from large-scale EHR sequences.<n>A unified embedding module encodes event identity, continuous values, units, and temporal attributes, while a Longformer-based encoder enables efficient modeling of long trajectories.
  arXiv  Detail & Related papers  (2025-11-27T08:10:52Z)
• Bridging Electronic Health Records and Clinical Texts: Contrastive Learning for Enhanced Clinical Tasks [0.0]
  We propose a framework that aligns the latent representations of structured EHR data with unstructured discharge summary notes.<n>It works by pulling together paired EHR and text embeddings while pushing apart unpaired ones.<n>Results demonstrate the effect of integrating domain knowledge from clinical notes into EHR-based pipelines.
  arXiv  Detail & Related papers  (2025-05-23T09:04:49Z)
• CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings [0.0]
  We introduce CAAT-EHR, a novel architecture designed to generate task-agnostic longitudinal embeddings from raw EHR data.<n>An autoregressive decoder complements the encoder by predicting future time points data during pre-training, ensuring that the resulting embeddings maintain temporal consistency and alignment.
  arXiv  Detail & Related papers  (2025-01-31T05:00:02Z)
• Toward Understanding BERT-Like Pre-Training for DNA Foundation Models [78.48760388079523]
  Existing pre-training methods for DNA sequences rely on direct adoptions of BERT pre-training from NLP. We introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pre-training by continuously expanding its mask boundary. RandomMask achieves a staggering 68.16% in Matthew's correlation coefficient for Epigenetic Mark Prediction, a groundbreaking increase of 19.85% over the baseline.
  arXiv  Detail & Related papers  (2023-10-11T16:40:57Z)
• UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation [93.88170217725805]
  We propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed. The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features. Our evaluations on five benchmarks, Synapse, BTCV, ACDC, BRaTs, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy.
  arXiv  Detail & Related papers  (2022-12-08T18:59:57Z)
• Prompt Tuning for Parameter-efficient Medical Image Segmentation [79.09285179181225]
  We propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets. We pre-train this architecture with a dedicated dense self-supervision scheme based on assignments to online generated prototypes. We demonstrate that the resulting neural network model is able to attenuate the gap between fully fine-tuned and parameter-efficiently adapted models.
  arXiv  Detail & Related papers  (2022-11-16T21:55:05Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• Longitudinal Variational Autoencoder [1.4680035572775534]
  A common approach to analyse high-dimensional data that contains missing values is to learn a low-dimensional representation using variational autoencoders (VAEs) Standard VAEs assume that the learnt representations are i.i.d., and fail to capture the correlations between the data samples. We propose the Longitudinal VAE (L-VAE), that uses a multi-output additive Gaussian process (GP) prior to extend the VAE's capability to learn structured low-dimensional representations. Our approach can simultaneously accommodate both time-varying shared and random effects, produce structured low-dimensional representations
  arXiv  Detail & Related papers  (2020-06-17T10:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.