Imputation of Unknown Missingness in Sparse Electronic Health Records

• URL: http://arxiv.org/abs/2602.20442v1
• Date: Tue, 24 Feb 2026 01:04:02 GMT
• Title: Imputation of Unknown Missingness in Sparse Electronic Health Records
• Authors: Jun Han, Josue Nassar, Sanjit Singh Batra, Aldo Cordova-Palomera, Vijay Nori, Robert E. Tillman,
• Abstract summary: We develop a general purpose algorithm for denoising data to recover unknown missing values in binary EHRs.<n>We design a transformer-based denoising neural network where the output is thresholded adaptively to recover values in cases where we predict data are missing.<n>Our results demonstrate improved accuracy in denoising medical codes within a real EHR dataset compared to existing imputation approaches.
• Score: 4.487420781682439
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning holds great promise for advancing the field of medicine, with electronic health records (EHRs) serving as a primary data source. However, EHRs are often sparse and contain missing data due to various challenges and limitations in data collection and sharing between healthcare providers. Existing techniques for imputing missing values predominantly focus on known unknowns, such as missing or unavailable values of lab test results; most do not explicitly address situations where it is difficult to distinguish what is missing. For instance, a missing diagnosis code in an EHR could signify either that the patient has not been diagnosed with the condition or that a diagnosis was made, but not shared by a provider. Such situations fall into the paradigm of unknown unknowns. To address this challenge, we develop a general purpose algorithm for denoising data to recover unknown missing values in binary EHRs. We design a transformer-based denoising neural network where the output is thresholded adaptively to recover values in cases where we predict data are missing. Our results demonstrate improved accuracy in denoising medical codes within a real EHR dataset compared to existing imputation approaches and leads to increased performance on downstream tasks using the denoised data. In particular, when applying our method to a real world application, predicting hospital readmission from EHRs, our method achieves statistically significant improvement over all existing baselines.

Related papers

• PRISM: Mitigating EHR Data Sparsity via Learning from Missing Feature Calibrated Prototype Patient Representations [7.075420686441701]
  PRISM is a framework that indirectly imputes data by leveraging prototype representations of similar patients.<n> PRISM also includes a feature confidence module, which evaluates the reliability of each feature considering missing statuses.<n>Our experiments on the MIMIC-III, MIMIC-IV, PhysioNet Challenge 2012, eICU datasets demonstrate PRISM's superior performance in predicting in-hospital mortality and 30-day readmission tasks.
  arXiv  Detail & Related papers  (2023-09-08T07:01:38Z)
• Time-dependent Iterative Imputation for Multivariate Longitudinal Clinical Data [0.0]
  Time-Dependent Iterative imputation offers a practical solution for imputing time-series data. When applied to a cohort consisting of more than 500,000 patient observations, our approach outperformed state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2023-04-16T16:10:49Z)
• Are we certain it's anomalous? [57.729669157989235]
  Anomaly detection in time series is a complex task since anomalies are rare due to highly non-linear temporal correlations. Here we propose the novel use of Hyperbolic uncertainty for Anomaly Detection (HypAD) HypAD learns self-supervisedly to reconstruct the input signal.
  arXiv  Detail & Related papers  (2022-11-16T21:31:39Z)
• MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models [78.72682320019737]
  We develop a general method, which we call MissDAG, to perform causal discovery from data with incomplete observations. MissDAG maximizes the expected likelihood of the visible part of observations under the expectation-maximization framework. We demonstrate the flexibility of MissDAG for incorporating various causal discovery algorithms and its efficacy through extensive simulations and real data experiments.
  arXiv  Detail & Related papers  (2022-05-27T09:59:46Z)
• To Impute or not to Impute? -- Missing Data in Treatment Effect Estimation [84.76186111434818]
  We identify a new missingness mechanism, which we term mixed confounded missingness (MCM), where some missingness determines treatment selection and other missingness is determined by treatment selection. We show that naively imputing all data leads to poor performing treatment effects models, as the act of imputation effectively removes information necessary to provide unbiased estimates. Our solution is selective imputation, where we use insights from MCM to inform precisely which variables should be imputed and which should not.
  arXiv  Detail & Related papers  (2022-02-04T12:08:31Z)
• Sequential Diagnosis Prediction with Transformer and Ontological Representation [35.88195694025553]
  We propose an end-to-end robust transformer-based model called SETOR to handle irregular intervals between a patient's visits with admitted timestamps and length of stay in each visit. Experiments conducted on two real-world healthcare datasets show that, our sequential diagnoses prediction model SETOR achieves better predictive results than previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-09-07T13:09:55Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Medical data wrangling with sequential variational autoencoders [5.9207487081080705]
  This paper proposes to model medical data records with heterogeneous data types and bursty missing data using sequential variational autoencoders (VAEs) We show that Shi-VAE achieves the best performance in terms of using both metrics, with lower computational complexity than the GP-VAE model.
  arXiv  Detail & Related papers  (2021-03-12T10:59:26Z)
• Handling Non-ignorably Missing Features in Electronic Health Records Data Using Importance-Weighted Autoencoders [8.518166245293703]
  We propose a novel extension of VAEs called Importance-Weighted Autoencoders (IWAEs) to flexibly handle Missing Not At Random patterns in the Physionet data. Our proposed method models the missingness mechanism using an embedded neural network, eliminating the need to specify the exact form of the missingness mechanism a priori.
  arXiv  Detail & Related papers  (2021-01-18T22:53:29Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)
• VAEs in the Presence of Missing Data [6.397263087026567]
  We develop a novel latent variable model of a corruption process which generates missing data, and derive a corresponding tractable evidence lower bound (ELBO) Our model is straightforward to implement, can handle both missing completely at random (MCAR) and missing not at random (MNAR) data, scales to high dimensional inputs and gives both the VAE encoder and decoder access to indicator variables for whether a data element is missing or not. On the MNIST and SVHN datasets we demonstrate improved marginal log-likelihood of observed data and better missing data imputation, compared to existing approaches.
  arXiv  Detail & Related papers  (2020-06-09T14:40:00Z)
• DeepEnroll: Patient-Trial Matching with Deep Embedding and Entailment Prediction [67.91606509226132]
  Clinical trials are essential for drug development but often suffer from expensive, inaccurate and insufficient patient recruitment. DeepEnroll is a cross-modal inference learning model to jointly encode enrollment criteria (tabular data) into a shared latent space for matching inference.
  arXiv  Detail & Related papers  (2020-01-22T17:51:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.