Scorecards for Synthetic Medical Data Evaluation and Reporting

• URL: http://arxiv.org/abs/2406.11143v1
• Date: Mon, 17 Jun 2024 02:11:59 GMT
• Title: Scorecards for Synthetic Medical Data Evaluation and Reporting
• Authors: Ghada Zamzmi, Adarsh Subbaswamy, Elena Sizikova, Edward Margerrison, Jana Delfino, Aldo Badano,
• Abstract summary: The growing utilization of synthetic medical data (SMD) in training and testing AI-driven tools in healthcare requires a systematic framework for assessing its quality. Here, we outline an evaluation framework designed to meet the unique requirements of medical applications. We introduce the concept of scorecards, which can serve as comprehensive reports that accompany artificially generated datasets.
• Score: 2.8262986891348056
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing utilization of synthetic medical data (SMD) in training and testing AI-driven tools in healthcare necessitates a systematic framework for assessing SMD quality. The current lack of a standardized methodology to evaluate SMD, particularly in terms of its applicability in various medical scenarios, is a significant hindrance to its broader acceptance and utilization in healthcare applications. Here, we outline an evaluation framework designed to meet the unique requirements of medical applications, and introduce the concept of SMD scorecards, which can serve as comprehensive reports that accompany artificially generated datasets. This can help standardize evaluation and enable SMD developers to assess and further enhance the quality of SMDs by identifying areas in need of attention and ensuring that the synthetic data more accurately approximate patient data.

Related papers

• Scoring Verifiers: Evaluating Synthetic Verification in Code and Reasoning [59.25951947621526]
  We introduce benchmarks designed to evaluate the impact of synthetic verification methods on assessing solution correctness. We analyze synthetic verification methods in standard, reasoning-based, and reward-based LLMs. Our results show that recent reasoning models significantly improve test case generation and that scaling test cases enhances verification accuracy.
  arXiv  Detail & Related papers  (2025-02-19T15:32:11Z)
• An Integrated Approach to AI-Generated Content in e-health [0.0]
  We propose an end-to-end class-conditioned framework to generate synthetic medical images and text data. Our framework integrates Diffusion and Large Language Models (LLMs) to generate data that closely match real-world patterns.
  arXiv  Detail & Related papers  (2025-01-18T14:35:29Z)
• RaTEScore: A Metric for Radiology Report Generation [59.37561810438641]
  This paper introduces a novel, entity-aware metric, as Radiological Report (Text) Evaluation (RaTEScore) RaTEScore emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. Our evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics, validated both on established public benchmarks and our newly proposed RaTE-Eval benchmark.
  arXiv  Detail & Related papers  (2024-06-24T17:49:28Z)
• A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models [3.672850225066168]
  generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data. Despite the potential benefits, concerns regarding privacy leakage have surfaced. We introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data.
  arXiv  Detail & Related papers  (2024-04-20T08:08:28Z)
• The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review [0.0]
  Development of trustworthy AI is especially important in medicine. We focus on the importance of data quality (training/test) in deep learning (DL) We propose the METRIC-framework, a specialised data quality framework for medical training data.
  arXiv  Detail & Related papers  (2024-02-21T09:15:46Z)
• Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare [33.855237079128955]
  In response to privacy concerns and regulatory requirements, using synthetic data has been suggested. We present a conceptual framework for quality assurance of SD for AI applications in healthcare. We propose stages necessary to support real-life applications.
  arXiv  Detail & Related papers  (2024-01-24T08:14:20Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)
• Evaluation of the Synthetic Electronic Health Records [3.255030588361125]
  This work outlines two metrics called Similarity and Uniqueness for sample-wise assessment of synthetic datasets. We demonstrate the proposed notions with several state-of-the-art generative models to synthesise Cystic Fibrosis (CF) patients' electronic health records.
  arXiv  Detail & Related papers  (2022-10-16T22:46:08Z)
• MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation [110.31526448744096]
  We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. We are building MedPerf, an open framework for benchmarking machine learning in the medical domain.
  arXiv  Detail & Related papers  (2021-09-29T18:09:41Z)
• Privacy-preserving medical image analysis [53.4844489668116]
  We present PriMIA, a software framework designed for privacy-preserving machine learning (PPML) in medical imaging. We show significantly better classification performance of a securely aggregated federated learning model compared to human experts on unseen datasets. We empirically evaluate the framework's security against a gradient-based model inversion attack.
  arXiv  Detail & Related papers  (2020-12-10T13:56:00Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.