Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images

• URL: http://arxiv.org/abs/2605.05522v1
• Date: Wed, 06 May 2026 23:46:46 GMT
• Title: Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images
• Abstract summary: We show two interacting failure modes in CT-to-MRI transfer.<n>Inefficient token usage caused by zero-padding to match pretrained input dimensions and ineffective feature adaptation.<n>Fine-tuning on identical rectal MRI datasets improved detection rates to 224/247 (90.7%) for SMIT and 219/247 (88.7%) for Swin UNETR.
• Score: 1.3958880590777554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretraining on large-scale datasets has been shown to improve transformer generalizability, even for out-of-domain (OOD) modalities and tasks. However, two common assumptions often fail under OOD transfer: that downstream datasets can be adapted to the fixed input geometry of pretrained models and that pretrained representations transfer effectively across imaging modalities. We show that these assumptions break down through two interacting failure modes in CT-to-MRI transfer: inefficient token usage caused by zero-padding to match pretrained input dimensions and ineffective feature adaptation. These failures led to accuracy degradation despite extensive fine-tuning. We investigated these failure modes using two CT-pretrained hierarchical shifted-window transformer backbones, SMIT and Swin UNETR, pretrained with different objectives and datasets. Mechanistic analysis introduced an attention dilution index (ADI), an entropy-based metric quantifying attention diverted toward uninformative padding tokens, and centered kernel alignment (CKA) to measure feature reuse in MRI tasks. ADI increased with zero-padding, while high feature reuse did not necessarily correspond to improved accuracy. To mitigate these issues, we introduced two interventions: a tumor-aware augmentation strategy to improve tumor appearance heterogeneity coverage and an anisotropic cropping strategy to restore token efficiency. Fine-tuning on identical rectal MRI datasets improved detection rates to 224/247 (90.7%) for SMIT and 219/247 (88.7%) for Swin UNETR, demonstrating improved robustness under CT-to-MRI transfer. This study is among the first to examine when pretrained transformers fail to transfer effectively across imaging modalities and how simple mitigation strategies, motivated by mechanistic analysis of datasets, can reduce transfer limitations while improving robustness and MRI detection.

Related papers

• Multi-Contrast MRI Motion Correction via Parameter-Informed Disentanglement and Adaptive Experts [56.74856091904914]
  We propose a unified framework combining parameter-informed contrast disentanglement with severity-aware adaptive correction.<n>ScanCLIP derives contrast embeddings from acquisition parameters to disentangle contrast style from anatomical content, yielding contrast-free features.<n>A Vision Transformer then estimates motion severity and routes features through a Mixture-of-Experts network, enabling targeted artifact correction.
  arXiv  Detail & Related papers  (2026-05-29T02:48:23Z)
• D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities [54.87947751720332]
  We propose a novel segmentation model D3Seg to maintain stable performance under missing-modality settings.<n>The proposed model achieves approximately 1.5-2.0% Dice improvement on enhancing tumor (ET) and around 1.0% on tumor core (TC) across multiple missing modality configurations.
  arXiv  Detail & Related papers  (2026-05-21T09:55:42Z)
• One Sequence to Segment Them All: Efficient Data Augmentation for CT and MRI Cross-Domain 3D Spine Segmentation [14.430939194449229]
  Deep learning-based medical image segmentation is increasingly used to support clinical diagnosis and develop new treatment strategies.<n>However, model performance remains limited by the scarcity of high-quality annotated data and insufficient generalization across imaging protocols.<n>This limitation is particularly evident in MRI and CT, where models are typically trained on a single acquisition sequence.
  arXiv  Detail & Related papers  (2026-05-04T19:26:29Z)
• Distilling Photon-Counting CT into Routine Chest CT through Clinically Validated Degradation Modeling [51.330452704464925]
  We train a latent diffusion model on 1,046 PCCTs and 405,379 EICTs from 145 hospitals.<n>We construct a large-scale dataset of over 17,316 publicly available EICTs enhanced to PCCT-like quality.<n>Our results suggest emerging imaging advances can be systematically distilled into routine EICT using limited high-quality scans as reference.
  arXiv  Detail & Related papers  (2026-04-08T17:47:35Z)
• Anatomical Token Uncertainty for Transformer-Guided Active MRI Acquisition [27.825869252861335]
  Full data acquisition in MRI is inherently slow, which limits clinical throughput and increases patient discomfort.<n>CSMRI seeks to accelerate acquisition by reconstructing images from under-sampled kspace data.
  arXiv  Detail & Related papers  (2026-03-23T10:49:23Z)
• Dual-Encoder Transformer-Based Multimodal Learning for Ischemic Stroke Lesion Segmentation Using Diffusion MRI [5.332404648315838]
  We study ischemic stroke lesion segmentation using multimodal diffusion MRI from the ISLES 2022 dataset.<n>Several state-of-the-art convolutional and transformer-based architectures, including U-Net variants, Swin-UNet, and TransUNet, are benchmarked.<n>Results show that transformer-based models outperform convolutional baselines, and the proposed dual-encoder TransUNet achieves the best performance, reaching a Dice score of 85.4% on the test set.
  arXiv  Detail & Related papers  (2025-12-23T15:24:31Z)
• Clinical Metadata Guided Limited-Angle CT Image Reconstruction [8.987752589028348]
  Limited-angle computed tomography (LACT) offers improved temporal resolution and reduced radiation dose for cardiac imaging, but suffers from severe artifacts due to truncated projections.<n>To address the ill-posedness of LACT reconstruction, we propose a two-stage diffusion framework guided by structured clinical metadata.
  arXiv  Detail & Related papers  (2025-09-01T20:14:15Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning [51.26601171361753]
  We propose ContextMRI, a text-conditioned diffusion model for MRI that integrates granular metadata into the reconstruction process.<n>We show that increasing the fidelity of metadata, ranging from slice location and contrast to patient age, sex, and pathology, systematically boosts reconstruction performance.
  arXiv  Detail & Related papers  (2025-01-08T05:15:43Z)
• Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning [6.44069573245889]
  Brain tumor segmentation remains a significant challenge, particularly in the context of multi-modal magnetic resonance imaging (MRI) We propose a novel strategy, which is called masked predicted pre-training, enabling robust feature learning from incomplete modality data. In the fine-tuning phase, we utilize a knowledge distillation technique to align features between complete and missing modality data, simultaneously enhancing model robustness.
  arXiv  Detail & Related papers  (2024-06-12T20:35:16Z)
• Weakly-supervised Biomechanically-constrained CT/MRI Registration of the Spine [72.85011943179894]
  We propose a weakly-supervised deep learning framework that preserves the rigidity and the volume of each vertebra while maximizing the accuracy of the registration. We specifically design these losses to depend only on the CT label maps since automatic vertebra segmentation in CT gives more accurate results contrary to MRI. Our results show that adding the anatomy-aware losses increases the plausibility of the inferred transformation while keeping the accuracy untouched.
  arXiv  Detail & Related papers  (2022-05-16T10:59:55Z)
• Negligible effect of brain MRI data preprocessing for tumor segmentation [36.89606202543839]
  We conduct experiments on three publicly available datasets and evaluate the effect of different preprocessing steps in deep neural networks. Our results demonstrate that most popular standardization steps add no value to the network performance. We suggest that image intensity normalization approaches do not contribute to model accuracy because of the reduction of signal variance with image standardization.
  arXiv  Detail & Related papers  (2022-04-11T17:29:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.