Cross-modal triage network: a multimodal deep learning framework for severity-based triage and visual explainability in chest radiographs

• URL: http://arxiv.org/abs/2609.04357v1
• Date: Thu, 03 Sep 2026 18:19:47 GMT
• Title: Cross-modal triage network: a multimodal deep learning framework for severity-based triage and visual explainability in chest radiographs
• Abstract summary: The CMTN was trained on 34,639 image-text pairs from MIMIC-CXR-JPG.<n>Heatmaps were evaluated against a blinded expert radiologist in a two-phase clinical audit.
• Score: 0.3167924351428519
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Purpose: Increased number of chest radiograph (CXR) scans create a triage bottleneck, queueing urgent examinations behind routine ones. Existing AI tools are predominantly unimodal binary classifiers lacking severity awareness, and multimodal systems are rarely benchmarked against expert radiologists. To this end, we developed a multimodal deep learning framework for joint severity triage, pathology detection, and native visual explanation. Approach: We propose the cross-modal triage network (CMTN), fusing a Swin Transformer V2 visual encoder with a PubMedBERT text encoder via gated cross-attention. The CMTN was trained on 34,639 image-text pairs (12,489 patients) from MIMIC-CXR-JPG, optimizing an ordinal focal loss for four-tier severity triage and binary cross-entropy for 14 pathologies. Beyond quantitative benchmarking, attention heatmaps were evaluated against a blinded expert radiologist in a two-phase clinical audit comparing model triage output to expert severity assessment (100 cases) and grading spatial-semantic concordance (116 heatmaps). Results: The CMTN achieved strong ordinal agreement with reference labels (quadratic weighted kappa [QWK] = 0.9341, 95\% CI: 0.9219 to 0.9449) and macro-AUROC of 0.9970 across 14 pathologies, with 34~ms latency, outperforming the state-of-the-art BioViL multimodal baseline (QWK = 0.7679). However, the blinded Phase I clinical audit revealed substantially lower agreement with genuine radiologist judgment (QWK = 0.1399). Phase II found 54.3\% of heatmaps achieved clinically acceptable spatial localization. Conclusions: The CMTN demonstrated an efficient multimodal architecture for CXR triage. The divergence between algorithmic and radiologist agreement demonstrates that benchmark performance against NLP-derived labels is insufficient, highlighting the need for radiologist-labeled ground truth before clinical deployment.

Related papers

• Auditable agentic AI for evidence-grounded thyroid ultrasound diagnosis and reporting [59.709162055944915]
  ThyroidXAgent is an agentic AI system that coordinates specialized diagnostic tools and stores their outputs as an auditable case-level evidence record.<n>It was evaluated on 28,458 non-overlapping test cases, including 8,721 cases from 35 centres in the private NHC-MISD-TUS cohort.<n>It achieved a mean Dice score of 87.21 percent for segmentation and a mean AUROC of 0.9466 for benign-malignant classification.
  arXiv  Detail & Related papers  (2026-08-12T21:02:59Z)
• A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT [15.152662172359355]
  Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization.<n>It carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload.<n>We introduce a novel multi-center benchmark for multi-organ abdominal disease diagnosis and automated radiology report generation.
  arXiv  Detail & Related papers  (2026-06-15T17:29:32Z)
• TriALS: Triphasic-Aided Liver Lesion Segmentation Benchmark in Non-Contrast CT [49.93685997940356]
  We describe the TriALS challenge for automated liver lesion segmentation under contrast-limited conditions.<n>The top-performing method achieved a mean venous-phase Dice of 0.754, consistent with human-level performance, yet dropped to 0.57 on NCCT.<n>Algorithm performance was most strongly predicted by training data scale and pre-training strategy.
  arXiv  Detail & Related papers  (2026-05-15T19:23:35Z)
• Radiology Report Generation with Layer-Wise Anatomical Attention [35.18016233072556]
  We introduce a compact image-to-text architecture that generates Findings section of chest X-ray reports.<n>The model combines a frozen Self-Distillation with No Labels v3 (DINOv3) Vision Transformer (ViT) encoder with a Generative Pre-trained Transformer 2 (GPT-2) decoder.
  arXiv  Detail & Related papers  (2025-12-18T18:17:57Z)
• Neural Discrete Representation Learning for Sparse-View CBCT Reconstruction: From Algorithm Design to Prospective Multicenter Clinical Evaluation [64.42236775544579]
  Cone beam computed tomography (CBCT)-guided puncture has become an established approach for diagnosing and treating thoracic tumours.<n>DeepPriorCBCT is a three-stage deep learning framework that achieves diagnostic-grade reconstruction using only one-sixth of the conventional radiation dose.
  arXiv  Detail & Related papers  (2025-11-30T12:45:02Z)
• Hemorica: A Comprehensive CT Scan Dataset for Automated Brain Hemorrhage Classification, Segmentation, and Detection [0.749500254646884]
  Hemorica is a publicly available collection of 372 head CT examinations acquired between 2012 and 2024.<n>Each scan has been exhaustively annotated for five ICH subtypes-epidural (EPH), subdural (SDH), subarachnoid (SAH), intraparenchymal (IPH)<n>Hemorica offers a unified, fine-grained benchmark that supports multi-task and curriculum learning.
  arXiv  Detail & Related papers  (2025-09-26T23:09:41Z)
• Benchmark of Segmentation Techniques for Pelvic Fracture in CT and X-ray: Summary of the PENGWIN 2024 Challenge [20.058385954540082]
  The PENGWIN challenge aimed to advance automated fracture segmentation.<n>The top-performing CT algorithm achieved an average fragment-wise intersection over union (IoU) of 0.930, demonstrating satisfactory accuracy.<n>The best algorithm attained an IoU of 0.774, highlighting the greater challenges posed by overlapping anatomical structures.
  arXiv  Detail & Related papers  (2025-04-03T08:19:36Z)
• Domain Transfer Through Image-to-Image Translation for Uncertainty-Aware Prostate Cancer Classification [42.75911994044675]
  We present a novel approach for unpaired image-to-image translation of prostate MRIs and an uncertainty-aware training approach for classifying clinically significant PCa. Our approach involves a novel pipeline for translating unpaired 3.0T multi-parametric prostate MRIs to 1.5T, thereby augmenting the available training data. Our experiments demonstrate that the proposed method significantly improves the Area Under ROC Curve (AUC) by over 20% compared to the previous work.
  arXiv  Detail & Related papers  (2023-07-02T05:26:54Z)
• Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling [66.75096111651062]
  We created a large-scale dataset of 10,021 thoracic CTs with 157 labels. We applied an ensemble of 3D anatomy segmentation models to extract anatomical pseudo-labels. Our resulting segmentation models demonstrated remarkable performance on CXR.
  arXiv  Detail & Related papers  (2023-06-06T18:01:08Z)
• Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification [52.77024349608834]
  To investigate chest radiograph (CXR) classification performance of vision transformers (ViT) and interpretability of attention-based saliency. ViTs were fine-tuned for lung disease classification using four public data sets: CheXpert, Chest X-Ray 14, MIMIC CXR, and VinBigData. ViTs had comparable CXR classification AUCs compared with state-of-the-art CNNs.
  arXiv  Detail & Related papers  (2023-03-03T12:05:41Z)
• A Global Benchmark of Algorithms for Segmenting Late Gadolinium-Enhanced Cardiac Magnetic Resonance Imaging [90.29017019187282]
  " 2018 Left Atrium Challenge" using 154 3D LGE-MRIs, currently the world's largest cardiac LGE-MRI dataset. Analyse of the submitted algorithms using technical and biological metrics was performed. Results show the top method achieved a dice score of 93.2% and a mean surface to a surface distance of 0.7 mm.
  arXiv  Detail & Related papers  (2020-04-26T08:49:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.