Cohort-attention Evaluation Metric against Tied Data: Studying Performance of Classification Models in Cancer Detection

• URL: http://arxiv.org/abs/2503.12755v1
• Date: Mon, 17 Mar 2025 02:50:40 GMT
• Title: Cohort-attention Evaluation Metric against Tied Data: Studying Performance of Classification Models in Cancer Detection
• Authors: Longfei Wei, Fang Sheng, Jianfei Zhang,
• Abstract summary: We propose the Cohort-Attention Evaluation Metrics (CAT) framework to address these challenges.<n>CAT introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity.<n>This approach enhances predictive reliability, fairness, and interpretability, providing a robust evaluation method for AI-driven medical screening models.
• Score: 1.3767986497772466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial intelligence (AI) has significantly improved medical screening accuracy, particularly in cancer detection and risk assessment. However, traditional classification metrics often fail to account for imbalanced data, varying performance across cohorts, and patient-level inconsistencies, leading to biased evaluations. We propose the Cohort-Attention Evaluation Metrics (CAT) framework to address these challenges. CAT introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity. Key metrics like CATSensitivity (CATSen), CATSpecificity (CATSpe), and CATMean ensure balanced and fair evaluation across diverse populations. This approach enhances predictive reliability, fairness, and interpretability, providing a robust evaluation method for AI-driven medical screening models.

Related papers

• Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection [1.4624458429745086]
  Current practices often rely on inconsistent and biased metrics.<n>Expert-level claims about AI performance are frequently made without rigorous validation.<n>This study proposes best practices tailored to the specific challenges of neonatal seizure detection.
  arXiv  Detail & Related papers  (2025-08-06T21:55:28Z)
• Advancements In Heart Disease Prediction: A Machine Learning Approach For Early Detection And Risk Assessment [0.0]
  This paper comprehends, assess, and analyze the role, relevance, and efficiency of machine learning models in predicting heart disease risks using clinical data. The Support Vector Machine (SVM) demonstrates the highest accuracy at 91.51%, confirming its superiority among the evaluated models in terms of predictive capability.
  arXiv  Detail & Related papers  (2024-10-16T22:32:19Z)
• From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
  We discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time. We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Explainable AI for Malnutrition Risk Prediction from m-Health and Clinical Data [3.093890460224435]
  This paper presents a novel AI framework for early and explainable malnutrition risk detection based on heterogeneous m-health data. We performed an extensive model evaluation including both subject-independent and personalised predictions. We also investigated several benchmark XAI methods to extract global model explanations.
  arXiv  Detail & Related papers  (2023-05-31T08:07:35Z)
• Auditing ICU Readmission Rates in an Clinical Database: An Analysis of Risk Factors and Clinical Outcomes [0.0]
  This study presents a machine learning pipeline for clinical data classification in the context of a 30-day readmission problem. The fairness audit uncovers disparities in equal opportunity, predictive parity, false positive rate parity, and false negative rate parity criteria. The study suggests the need for collaborative efforts among researchers, policymakers, and practitioners to address bias and fairness in artificial intelligence (AI) systems.
  arXiv  Detail & Related papers  (2023-04-12T17:09:38Z)
• Evaluating Probabilistic Classifiers: The Triptych [62.997667081978825]
  We propose and study a triptych of diagnostic graphics that focus on distinct and complementary aspects of forecast performance. The reliability diagram addresses calibration, the receiver operating characteristic (ROC) curve diagnoses discrimination ability, and the Murphy diagram visualizes overall predictive performance and value.
  arXiv  Detail & Related papers  (2023-01-25T19:35:23Z)
• Towards Reliable Medical Image Segmentation by utilizing Evidential Calibrated Uncertainty [52.03490691733464]
  We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks. By leveraging subjective logic theory, we explicitly model probability and uncertainty for the problem of medical image segmentation. DeviS incorporates an uncertainty-aware filtering module, which utilizes the metric of uncertainty-calibrated error to filter reliable data.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)
• Predictive Modeling of ICU Healthcare-Associated Infections from Imbalanced Data. Using Ensembles and a Clustering-Based Undersampling Approach [55.41644538483948]
  This work is focused on both the identification of risk factors and the prediction of healthcare-associated infections in intensive-care units. The aim is to support decision making addressed at reducing the incidence rate of infections.
  arXiv  Detail & Related papers  (2020-05-07T16:13:12Z)
• Uncertainty estimation for classification and risk prediction on medical tabular data [0.0]
  This work advances the understanding of uncertainty estimation for classification and risk prediction on medical data. In a data-scarce field such as healthcare, the ability to measure the uncertainty of a model's prediction could potentially lead to improved effectiveness of decision support tools.
  arXiv  Detail & Related papers  (2020-04-13T08:46:41Z)
• Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions [48.91284724066349]
  Off-policy evaluation in reinforcement learning offers the chance of using observational data to improve future outcomes in domains such as healthcare and education. Traditional measures such as confidence intervals may be insufficient due to noise, limited data and confounding. We develop a method that could serve as a hybrid human-AI system, to enable human experts to analyze the validity of policy evaluation estimates.
  arXiv  Detail & Related papers  (2020-02-10T00:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.