Estimators of Entropy and Information via Inference in Probabilistic Models

• URL: http://arxiv.org/abs/2202.12363v1
• Date: Thu, 24 Feb 2022 21:04:33 GMT
• Title: Estimators of Entropy and Information via Inference in Probabilistic Models
• Abstract summary: Estimating information-theoretic quantities such as entropy and mutual information is central to many problems in statistics and machine learning. This paper presents estimators of entropy via inference (EEVI), which deliver upper and lower bounds on many information quantities for arbitrary variables.
• Score: 1.4502611532302037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating information-theoretic quantities such as entropy and mutual information is central to many problems in statistics and machine learning, but challenging in high dimensions. This paper presents estimators of entropy via inference (EEVI), which deliver upper and lower bounds on many information quantities for arbitrary variables in a probabilistic generative model. These estimators use importance sampling with proposal distribution families that include amortized variational inference and sequential Monte Carlo, which can be tailored to the target model and used to squeeze true information values with high accuracy. We present several theoretical properties of EEVI and demonstrate scalability and efficacy on two problems from the medical domain: (i) in an expert system for diagnosing liver disorders, we rank medical tests according to how informative they are about latent diseases, given a pattern of observed symptoms and patient attributes; and (ii) in a differential equation model of carbohydrate metabolism, we find optimal times to take blood glucose measurements that maximize information about a diabetic patient's insulin sensitivity, given their meal and medication schedule.

Related papers

• Explainable Retinal Imaging for Prediction of Multi-Organ Dysfunction in Type 2 Diabetes [2.153295259565971]
  System-level abnormality indices were constructed to quantify organ-specific dysfunction.<n>Supervised machine learning models were trained to predict multi-system dysregulation.<n>Hyperglycaemia, renal impairment, dyslipidaemia, and inflammation were the dominant drivers of multi-system risk.
  arXiv  Detail & Related papers  (2026-05-24T07:31:57Z)
• Tailoring Adverse Event Prediction in Type 1 Diabetes with Patient-Specific Deep Learning Models [4.201695852847293]
  This paper presents a deep learning-based approach for personalized blood glucose prediction.<n>Unlike traditional generalized models, our method accounts for individual variability, enabling more effective subject-specific predictions.<n>Results show that personalized models significantly improve the prediction of adverse events.
  arXiv  Detail & Related papers  (2026-01-21T11:57:51Z)
• Beyond Traditional Diagnostics: Transforming Patient-Side Information into Predictive Insights with Knowledge Graphs and Prototypes [55.310195121276074]
  We propose a Knowledge graph-enhanced, Prototype-aware, and Interpretable (KPI) framework to predict diseases.<n>It integrates structured and trusted medical knowledge into a unified disease knowledge graph, constructs clinically meaningful disease prototypes, and employs contrastive learning to enhance predictive accuracy.<n>It provides clinically valid explanations that closely align with patient narratives, highlighting its practical value for patient-centered healthcare delivery.
  arXiv  Detail & Related papers  (2025-12-09T05:37:54Z)
• Semi-supervised Clustering Through Representation Learning of Large-scale EHR Data [5.591260685112265]
  SCORE is a semi-supervised representation learning framework that captures multi-domain disease profiles through patient embeddings.<n>To handle the computational challenges of large-scale data, it introduces a hybrid Expectation-Maximization (EM) and Gaussian Variational Approximation (GVA) algorithm.<n>Our analysis shows that incorporating unlabeled data enhances accuracy and reduces sensitivity to label scarcity.
  arXiv  Detail & Related papers  (2025-05-27T05:20:17Z)
• A Comparative Study on Machine Learning Models to Classify Diseases Based on Patient Behaviour and Habits [0.0]
  This study uses six supervised ML approaches to investigate the correlation between PRF and Diabetes, Stroke, Heart Disease (HD), and Kidney Disease (KD) It aims to compare and evaluate ML algorithms for classifying HD based on PRF as well as Diabetes, Stroke, Asthma, Skin Cancer, and KD as attributes.
  arXiv  Detail & Related papers  (2024-12-21T20:46:40Z)
• Effect of Clinical History on Predictive Model Performance for Renal Complications of Diabetes [1.4330510916280879]
  Diabetes is a chronic disease characterised by a high risk of developing diabetic nephropathy. The early identification of individuals at heightened risk of such complications or their exacerbation can be of paramount importance to set a correct course of treatment. We develop an array of logistic regression models to predict, over different prediction horizons, the crossing of clinically relevant glomerular filtration rate (eGFR) thresholds for patients with diabetes.
  arXiv  Detail & Related papers  (2024-09-10T20:27:00Z)
• Deep State-Space Generative Model For Correlated Time-to-Event Predictions [54.3637600983898]
  We propose a deep latent state-space generative model to capture the interactions among different types of correlated clinical events. Our method also uncovers meaningful insights about the latent correlations among mortality and different types of organ failures.
  arXiv  Detail & Related papers  (2024-07-28T02:42:36Z)
• Interpretable Causal Inference for Analyzing Wearable, Sensor, and Distributional Data [62.56890808004615]
  We develop an interpretable method for distributional data analysis that ensures trustworthy and robust decision-making. We demonstrate ADD MALTS' utility by studying the effectiveness of continuous glucose monitors in mitigating diabetes risks.
  arXiv  Detail & Related papers  (2023-12-17T00:42:42Z)
• Unsupervised EHR-based Phenotyping via Matrix and Tensor Decompositions [0.6875312133832078]
  We provide a comprehensive review of low-rank approximation-based approaches for computational phenotyping. Recent developments have adapted low-rank data approximation methods by incorporating different constraints and regularizations that facilitate interpretability further.
  arXiv  Detail & Related papers  (2022-09-01T09:47:27Z)
• Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection [7.263558963357268]
  Unpredictable ML model behavior on unseen data, especially in the health domain, raises serious concerns about its safety. We show that Mahalanobis distance- and Gram matrices-based out-of-distribution detection methods are able to detect out-of-distribution data with high accuracy. We then translate the out-of-distribution score into a human interpretable CONFIDENCE SCORE to investigate its effect on the users' interaction with health ML applications.
  arXiv  Detail & Related papers  (2021-10-26T20:49:01Z)
• Pandemic model with data-driven phase detection, a study using COVID-19 data [0.0]
  The recent COVID-19 pandemic has promoted vigorous scientific activity in an effort to understand, advice and control the pandemic. Data is now freely available at a staggering rate worldwide. Health officials have recognized the need for more accurate models that can adjust to sudden changes.
  arXiv  Detail & Related papers  (2021-10-24T14:07:32Z)
• Variational Knowledge Distillation for Disease Classification in Chest X-Rays [102.04931207504173]
  We propose itvariational knowledge distillation (VKD), which is a new probabilistic inference framework for disease classification based on X-rays. We demonstrate the effectiveness of our method on three public benchmark datasets with paired X-ray images and EHRs.
  arXiv  Detail & Related papers  (2021-03-19T14:13:56Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)
• Personalized pathology test for Cardio-vascular disease: Approximate Bayesian computation with discriminative summary statistics learning [48.7576911714538]
  We propose a platelet deposition model and an inferential scheme to estimate the biologically meaningful parameters using approximate computation. This work opens up an unprecedented opportunity of personalized pathology test for CVD detection and medical treatment.
  arXiv  Detail & Related papers  (2020-10-13T15:20:21Z)
• A General Framework for Survival Analysis and Multi-State Modelling [70.31153478610229]
  We use neural ordinary differential equations as a flexible and general method for estimating multi-state survival models. We show that our model exhibits state-of-the-art performance on popular survival data sets and demonstrate its efficacy in a multi-state setting.
  arXiv  Detail & Related papers  (2020-06-08T19:24:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.