Regional Attention-Enhanced Swin Transformer for Clinically Relevant Medical Image Captioning

• URL: http://arxiv.org/abs/2511.09893v1
• Date: Fri, 14 Nov 2025 01:16:58 GMT
• Title: Regional Attention-Enhanced Swin Transformer for Clinically Relevant Medical Image Captioning
• Abstract summary: We present a Swin-BART encoder-decoder system with a lightweight regional attention module.<n>Our model achieves state-of-the-art semantic fidelity while remaining compact and interpretable.<n>The proposed design yields accurate, clinically phrased captions and transparent regional attributions.
• Score: 4.754208048304996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated medical image captioning translates complex radiological images into diagnostic narratives that can support reporting workflows. We present a Swin-BART encoder-decoder system with a lightweight regional attention module that amplifies diagnostically salient regions before cross-attention. Trained and evaluated on ROCO, our model achieves state-of-the-art semantic fidelity while remaining compact and interpretable. We report results as mean$\pm$std over three seeds and include $95\%$ confidence intervals. Compared with baselines, our approach improves ROUGE (proposed 0.603, ResNet-CNN 0.356, BLIP2-OPT 0.255) and BERTScore (proposed 0.807, BLIP2-OPT 0.645, ResNet-CNN 0.623), with competitive BLEU, CIDEr, and METEOR. We further provide ablations (regional attention on/off and token-count sweep), per-modality analysis (CT/MRI/X-ray), paired significance tests, and qualitative heatmaps that visualize the regions driving each description. Decoding uses beam search (beam size $=4$), length penalty $=1.1$, $no\_repeat\_ngram\_size$ $=3$, and max length $=128$. The proposed design yields accurate, clinically phrased captions and transparent regional attributions, supporting safe research use with a human in the loop.

Related papers

• Cross-modal triage network: a multimodal deep learning framework for severity-based triage and visual explainability in chest radiographs [0.3167924351428519]
  The CMTN was trained on 34,639 image-text pairs from MIMIC-CXR-JPG.<n>Heatmaps were evaluated against a blinded expert radiologist in a two-phase clinical audit.
  arXiv  Detail & Related papers  (2026-09-03T18:19:47Z)
• Text-Guided Refinement of Multi-sequence Glioma Subregion Segmentation with a Vision-Language Foundation Model [5.368430154049423]
  3D vision-language foundation model can perform instruction-guided refinement of glioma subregion segmentations.<n> sensitivity to correct, blank, and contradictory prompts suggests text-dependent contour editing rather than nonspecific post-processing.
  arXiv  Detail & Related papers  (2026-08-05T20:17:02Z)
• RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding [14.036871627954156]
  We introduce RadPRISM, which makes a clinician-defined radiology schema a designated stratification axis.<n>An on-premise large language model extracts per-concept text spans from free-text reports, and each clinical concept is aligned in its own dedicated visual subspace.
  arXiv  Detail & Related papers  (2026-07-31T16:14:24Z)
• Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance [16.566888073449714]
  Vision--language models (VLMs) for radiology report generation (RRG) can produce long-form chest CT reports from volumetric scans.<n>Existing methods face two key limitations: (i) training supervision is often coarse, aligning a whole CT volume with a full free-text report without explicit alignment for fine-grained attributes or pathology locations.<n>We propose emphDiscrimi Cue-Prompting with Prompt Dropout (DCP-PD), a plug-and-play framework that distills fine-grained cues from free-text reports and uses them to guide report generation.
  arXiv  Detail & Related papers  (2026-04-12T03:25:41Z)
• WristMIR: Coarse-to-Fine Region-Aware Retrieval of Pediatric Wrist Radiographs with Radiology Report-Driven Learning [0.39146761527401425]
  WristMIR is a region-aware pediatric wrist radiograph retrieval framework.<n>It learns fine-grained, clinically meaningful image representations without any manual image-level annotations.
  arXiv  Detail & Related papers  (2026-02-08T08:57:57Z)
• MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification [2.748370039701545]
  We present MedicalPatchNet, an inherently self-explainable architecture for chest X-ray classification.<n>MedicalPatchNet splits images into non-overlapping patches, independently classifies each patch, and aggregates predictions.<n>By providing explicit, reliable explanations even accessible to non-AI experts, MedicalPatchNet mitigates risks associated with shortcut learning.
  arXiv  Detail & Related papers  (2025-09-09T08:02:10Z)
• A Novel Attention-Augmented Wavelet YOLO System for Real-time Brain Vessel Segmentation on Transcranial Color-coded Doppler [49.03919553747297]
  We propose an AI-powered, real-time CoW auto-segmentation system capable of efficiently capturing cerebral arteries.<n>No prior studies have explored AI-driven cerebrovascular segmentation using Transcranial Color-coded Doppler (TCCD)<n>The proposed AAW-YOLO demonstrated strong performance in segmenting both ipsilateral and contralateral CoW vessels.
  arXiv  Detail & Related papers  (2025-08-19T14:41:22Z)
• A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports [2.629902609930227]
  We introduce Med-CTX, a fully transformer based framework for explainable breast cancer ultrasound segmentation.<n>We integrate clinical radiology reports to boost both performance and interpretability.<n>On the BUS-BRA dataset, Med-CTX achieves a Dice score of 99% and an IoU of 95%, beating existing baselines U-Net, ViT, and Swin.
  arXiv  Detail & Related papers  (2025-08-19T12:55:10Z)
• Explainable Anatomy-Guided AI for Prostate MRI: Foundation Models and In Silico Clinical Trials for Virtual Biopsy-based Risk Assessment [3.5408411348831232]
  We present a fully automated, anatomically guided deep learning pipeline for prostate cancer (PCa) risk stratification using routine MRI.<n>The pipeline integrates three key components: an nnU-Net module for segmenting the prostate gland and its zones on axial T2-weighted MRI; a classification module based on the DiceedPT Swin Transformer foundation model, fine-tuned on 3D patches with optional anatomical priors and clinical data; and a VAE-GAN framework for generating counterfactual heatmaps that localize decision-driving image regions.
  arXiv  Detail & Related papers  (2025-05-23T14:40:09Z)
• A simple and effective approach for body part recognition on CT scans based on projection estimation [3.5436187733613087]
  This study proposes a simple, yet effective approach based on 2D X-ray-like estimation of 3D CT scans for body region identification.<n>In the proposed approach, estimated 2D images were utilized to identify 14 distinct body regions, providing valuable information for constructing a high-quality medical dataset.
  arXiv  Detail & Related papers  (2025-04-30T17:13:44Z)
• Cancer-Net PCa-Seg: Benchmarking Deep Learning Models for Prostate Cancer Segmentation Using Synthetic Correlated Diffusion Imaging [65.83291923029985]
  Prostate cancer (PCa) is the most prevalent cancer among men in the United States, accounting for nearly 300,000 cases, 29% of all diagnoses and 35,000 total deaths in 2024.<n>Traditional screening methods such as prostate-specific antigen (PSA) testing and magnetic resonance imaging (MRI) have been pivotal in diagnosis, but have faced limitations in specificity and generalizability.<n>We employ several state-of-the-art deep learning models, including U-Net, SegResNet, Swin UNETR, Attention U-Net, and LightM-UNet, to segment prostate glands from a 200 CDI$
  arXiv  Detail & Related papers  (2025-01-15T22:23:41Z)
• Towards a Benchmark for Colorectal Cancer Segmentation in Endorectal Ultrasound Videos: Dataset and Model Development [59.74920439478643]
  In this paper, we collect and annotated the first benchmark dataset that covers diverse ERUS scenarios. Our ERUS-10K dataset comprises 77 videos and 10,000 high-resolution annotated frames. We introduce a benchmark model for colorectal cancer segmentation, named the Adaptive Sparse-context TRansformer (ASTR)
  arXiv  Detail & Related papers  (2024-08-19T15:04:42Z)
• CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios [53.94122089629544]
  We introduce CT-GLIP (Grounded Language-Image Pretraining with CT scans), a novel method that constructs organ-level image-text pairs to enhance multimodal contrastive learning. Our method, trained on a multimodal CT dataset comprising 44,011 organ-level vision-text pairs from 17,702 patients across 104 organs, demonstrates it can identify organs and abnormalities in a zero-shot manner using natural languages.
  arXiv  Detail & Related papers  (2024-04-23T17:59:01Z)
• Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification [52.77024349608834]
  To investigate chest radiograph (CXR) classification performance of vision transformers (ViT) and interpretability of attention-based saliency. ViTs were fine-tuned for lung disease classification using four public data sets: CheXpert, Chest X-Ray 14, MIMIC CXR, and VinBigData. ViTs had comparable CXR classification AUCs compared with state-of-the-art CNNs.
  arXiv  Detail & Related papers  (2023-03-03T12:05:41Z)
• ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax [5.168314889999992]
  We propose a novel vision-language model, ConTEXTual Net, for the task of pneumothorax segmentation on chest radiographs. We trained it on the CANDID-PTX dataset consisting of 3,196 positive cases of pneumothorax. It achieved a Dice score of 0.716$pm$0.016, which was similar to the degree of inter-reader variability. It outperformed both vision-only models and a competing vision-language model.
  arXiv  Detail & Related papers  (2023-03-02T22:36:19Z)
• Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays [65.88435151891369]
  Radiomics-Guided Transformer (RGT) fuses textitglobal image information with textitlocal knowledge-guided radiomics information. RGT consists of an image Transformer branch, a radiomics Transformer branch, and fusion layers that aggregate image and radiomic information.
  arXiv  Detail & Related papers  (2022-07-10T06:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.