REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference

• URL: http://arxiv.org/abs/2606.07141v1
• Date: Fri, 05 Jun 2026 10:51:20 GMT
• Title: REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference
• Abstract summary: We introduce REMEDI, an extensive benchmark for machine unlearning tailored to multi-label and multiclass clinical disease inference.<n> REMEDI considers: (1) a relevant application domain (medical), (2) comprehensive unlearning setups involving diverse sets of forget instances, (3) challenging unlearning scenarios including multi-label and multi-class classification tasks, and (4) evaluation metrics involving performance both in terms of utility and extent of unlearning achieved.
• Score: 15.89997509050996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models trained for clinical disease inference are trained on patient data, which may include sensitive and private information, and data owners may request the removal of their data from a trained model due to privacy or copyright concerns. However, exactly unlearning patient-specific data is intractable, and retraining with minor data removal is resource-intensive. While there exists several machine unlearning methods that can be used, their utility is generally restricted to non-medical domains. Moreover, the existing benchmarks for evaluating such unlearning methods primarily utilize synthetically curated datasets, which are not truly representative of real-world systems. Hence, the effectiveness of these unlearning methods in the medical domain is largely unclear. To this end, we introduce REMEDI, an extensive benchmark for machine unlearning tailored to multi-label and multiclass clinical disease inference, where label correlations, longitudinal structure, and safety constraints make unlearning particularly challenging. Unlike the existing benchmarks, REMEDI considers: (1) a relevant application domain (medical), (2) comprehensive unlearning setups involving diverse sets of forget instances, (3) challenging unlearning scenarios including multi-label and multi-class classification tasks, and (4) evaluation metrics involving performance both in terms of utility and extent of unlearning achieved. REMEDI is developed using the MIMIC-III clinical database that contains comprehensive clinical data of patients. Experiments with existing unlearning methods indicate that there exists a trade-off between utility and unlearning performance. They are also largely unsuited to multi-label classification tasks. To facilitate reproducibility, we make our benchmark publicly available.

Related papers

• AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis [10.933595109116768]
  We introduce AMNESIA, the first large-scale, open source benchmark for medical unlearning.<n>We use it to evaluate four widely used unlearning methods at both random patient and disease-level.<n>We show that unlearning individual patients erodes knowledge of others with the same condition, calling for methods that can better separate patients from shared clinical knowledge.
  arXiv  Detail & Related papers  (2026-05-28T21:46:46Z)
• MedForget: Hierarchy-Aware Multimodal Unlearning Testbed for Medical AI [66.0701326117134]
  MedForget is a hierarchy-aware multimodal unlearning testbed for building compliant medical AI systems.<n>We show that existing methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance.<n>We introduce a reconstruction attack that progressively adds hierarchical level context to prompts.
  arXiv  Detail & Related papers  (2025-12-10T17:55:06Z)
• Handling missing values in clinical machine learning: Insights from an expert study [10.637366819633302]
  Inherently interpretable machine learning (IML) models offer valuable support for clinical decision-making.<n>Traditional approaches, such as imputation or discarding incomplete records, are often impractical in scenarios where data is missing at test time.<n>We surveyed 55 clinicians from 29 French trauma centers to study their interaction with three IML models.
  arXiv  Detail & Related papers  (2024-11-14T17:02:41Z)
• An Efficient Contrastive Unimodal Pretraining Method for EHR Time Series Data [35.943089444017666]
  We propose an efficient method of contrastive pretraining tailored for long clinical timeseries data. Our model demonstrates the ability to impute missing measurements, providing clinicians with deeper insights into patient conditions.
  arXiv  Detail & Related papers  (2024-10-11T19:05:25Z)
• Active Foundational Models for Fault Diagnosis of Electrical Motors [0.5999777817331317]
  Fault detection and diagnosis of electrical motors is of utmost importance in ensuring the safe and reliable operation of industrial systems. The existing data-driven deep learning approaches for machine fault diagnosis rely extensively on huge amounts of labeled samples. We propose a foundational model-based Active Learning framework that utilizes less amount of labeled samples.
  arXiv  Detail & Related papers  (2023-11-27T03:25:12Z)
• Improving Multiple Sclerosis Lesion Segmentation Across Clinical Sites: A Federated Learning Approach with Noise-Resilient Training [75.40980802817349]
  Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. We introduce a Decoupled Hard Label Correction (DHLC) strategy that considers the imbalanced distribution and fuzzy boundaries of MS lesions. We also introduce a Centrally Enhanced Label Correction (CELC) strategy, which leverages the aggregated central model as a correction teacher for all sites.
  arXiv  Detail & Related papers  (2023-08-31T00:36:10Z)
• LifeLonger: A Benchmark for Continual Disease Classification [59.13735398630546]
  We introduce LifeLonger, a benchmark for continual disease classification on the MedMNIST collection. Task and class incremental learning of diseases address the issue of classifying new samples without re-training the models from scratch. Cross-domain incremental learning addresses the issue of dealing with datasets originating from different institutions while retaining the previously obtained knowledge.
  arXiv  Detail & Related papers  (2022-04-12T12:25:05Z)
• Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases [57.90226879210227]
  FedCy is a semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos. We demonstrate significant performance gains over state-of-the-art FSSL methods on the task of automatic recognition of surgical phases.
  arXiv  Detail & Related papers  (2022-03-14T17:44:53Z)
• Active learning for medical code assignment [55.99831806138029]
  We demonstrate the effectiveness of Active Learning (AL) in multi-label text classification in the clinical domain. We apply a set of well-known AL methods to help automatically assign ICD-9 codes on the MIMIC-III dataset. Our results show that the selection of informative instances provides satisfactory classification with a significantly reduced training set.
  arXiv  Detail & Related papers  (2021-04-12T18:11:17Z)
• Uncovering the structure of clinical EEG signals with self-supervised learning [64.4754948595556]
  Supervised learning paradigms are often limited by the amount of labeled data that is available. This phenomenon is particularly problematic in clinically-relevant data, such as electroencephalography (EEG) By extracting information from unlabeled data, it might be possible to reach competitive performance with deep neural networks.
  arXiv  Detail & Related papers  (2020-07-31T14:34:47Z)
• Real-World Multi-Domain Data Applications for Generalizations to Clinical Settings [1.508558791031741]
  Deep learning models perform well when trained on standardized datasets from artificial settings, such as clinical trials. We show that by employing a self-supervised approach with transfer learning on a multi-domain real-world dataset, we can achieve 16% relative improvement on a standardized dataset.
  arXiv  Detail & Related papers  (2020-07-24T17:41:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.