When Repository Labels Are Not Image-Level Truth: A Supervision Auditing Framework for Chest Radiograph AI

• URL: http://arxiv.org/abs/2608.10084v1
• Date: Mon, 10 Aug 2026 18:00:50 GMT
• Title: When Repository Labels Are Not Image-Level Truth: A Supervision Auditing Framework for Chest Radiograph AI
• Abstract summary: We introduce Repository Supervision Auditing (RSA), a framework that evaluates repository-derived labels against expert image-level annotations.<n> RSA compares repository labels with radiologist-reviewed image annotations, characterizes disagreement sources, and builds a curated cohort for deployment-oriented evaluation.<n>Most discrepancies resulted from non-mention rather than explicit report negation, with expert-confirmed cardiomegaly identified in nearly half of studies assigned a repository-derived No Finding label.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Public chest X-ray repositories are widely used to train medical AI systems, yet their labels are typically extracted from radiology reports rather than verified directly on images. As a result, repository labels are often treated as image-level ground truth without validating whether they reflect what is actually visible in the radiograph. We introduce Repository Supervision Auditing (RSA), a framework that evaluates repository-derived labels against expert image-level annotations before model development. Using cardiomegaly in MIMIC-CXR as a case study, RSA compares repository labels with radiologist-reviewed image annotations, characterizes disagreement sources, and builds a curated cohort for deployment-oriented evaluation. Repository-derived cardiomegaly labels showed near-zero agreement with expert image-level assessment, identifying only 1% of expert-confirmed cases. Most discrepancies resulted from non-mention rather than explicit report negation, with expert-confirmed cardiomegaly identified in nearly half of studies assigned a repository-derived No Finding label. Using the resulting expert-curated cohort, a DenseNet121 model achieved a test ROC-AUC of 0.853. These findings show that repository labels may not reliably represent image-level truth and highlight supervision auditing as a critical step for developing trustworthy medical imaging AI.

Related papers

• Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation [20.57244883423868]
  We propose PU-DPO, a preference optimization framework to prevent omission noise from corrupting the preference signal.<n>We show that PU-DPO yields consistent gains in detection rates and recovery of hidden positives across multiple pathologies.
  arXiv  Detail & Related papers  (2026-08-05T18:59:23Z)
• Phenotyping TPF via Self-Supervised Learning: A Label-Agnostic Framework with Expert Validation [2.201241345037463]
  We design, implement, and validate a label-agnostic framework to learn fracture representations directly from imaging data without observer-assigned labels.<n>A RadImageNet-pretrained ResNet-50 encoder is fine-tuned on 154 cleaned knee radiographs.<n>We show that label-agnostic SSL phenotyping is a reproducible and clinically interpretable complement to conventional classification.
  arXiv  Detail & Related papers  (2026-06-15T21:04:31Z)
• Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection [95.08316274158165]
  X-AIGD provides pixel-level, categorized annotations of perceptual artifacts, spanning low-level distortions, high-level semantics, and cognitive-level counterfactuals.<n>Existing AIGI detectors demonstrate negligible reliance on perceptual artifacts, even at the most basic distortion level.<n>Explicitly aligning model attention with artifact regions can increase the interpretability and generalization of detectors.
  arXiv  Detail & Related papers  (2026-01-27T10:09:17Z)
• RSNA Large Language Model Benchmark Dataset for Chest Radiographs of Cardiothoracic Disease: Radiologist Evaluation and Validation Enhanced by AI Labels (REVEAL-CXR) [27.699008063855146]
  GPT-4o extracted abnormal findings from 13,735 chest radiographs and their corresponding reports from the MIDRC.<n>1,000 were sampled on the basis of the AI-suggested benchmark labels for expert review.<n>17 chest radiologists participated, and they marked "Agree all", "Agree mostly" or "Disagree" to indicate their assessment of the correctness of the LLM suggested labels.
  arXiv  Detail & Related papers  (2026-01-21T16:04:01Z)
• Spatial-aware Attention Generative Adversarial Network for Semi-supervised Anomaly Detection in Medical Image [63.59114880750643]
  We introduce a novel Spatial-aware Attention Generative Adrialversa Network (SAGAN) for one-class semi-supervised generation of health images. SAGAN generates high-quality health images corresponding to unlabeled data, guided by the reconstruction of normal images and restoration of pseudo-anomaly images. Extensive experiments on three medical datasets demonstrate that the proposed SAGAN outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-05-21T15:41:34Z)
• Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays [65.88435151891369]
  Radiomics-Guided Transformer (RGT) fuses textitglobal image information with textitlocal knowledge-guided radiomics information. RGT consists of an image Transformer branch, a radiomics Transformer branch, and fusion layers that aggregate image and radiomic information.
  arXiv  Detail & Related papers  (2022-07-10T06:32:56Z)
• Cross-Modal Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop [63.81818077092879]
  We propose an end-to-end semi-supervised cross-modal contrastive learning framework for medical images. We first apply an image encoder to classify the chest X-rays and to generate the image features. The radiomic features are then passed through another dedicated encoder to act as the positive sample for the image features generated from the same chest X-ray.
  arXiv  Detail & Related papers  (2021-04-11T09:16:29Z)
• VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels [4.865330207715854]
  We show that radiologists labeling reports significantly disagree with radiologists labeling chest X-ray images. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images.
  arXiv  Detail & Related papers  (2021-02-23T03:02:36Z)
• Labelling imaging datasets on the basis of neuroradiology reports: a validation study [0.3871995016053975]
  We show that, in our experience, assigning binary labels to images from reports alone is highly accurate. In contrast to the binary labels, however, the accuracy of more granular labelling is dependent on the category. We also show that downstream model performance is reduced when labelling of training reports is performed by a non-specialist.
  arXiv  Detail & Related papers  (2020-07-08T16:12:10Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)
• Do Public Datasets Assure Unbiased Comparisons for Registration Evaluation? [96.53940048041248]
  We use the variogram to screen the manually annotated landmarks in two datasets used to benchmark registration in image-guided neurosurgeries. Using variograms, we identified potentially problematic cases and had them examined by experienced radiologists.
  arXiv  Detail & Related papers  (2020-03-20T20:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.