Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation

• URL: http://arxiv.org/abs/2605.16393v1
• Date: Tue, 12 May 2026 11:56:46 GMT
• Title: Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation
• Abstract summary: We introduce ViTC-UNet, which conditions a UNet on frozen pre-trained ViT representations through learnable tokens and a two-way attention decoder.<n>This combines ViT global visual priors with the local inductive bias and high-resolution decoding capacity of UNets.<n>ViTC-UNet outperforms baseline results in semantic segmentation tasks across MRI and CT modalities.
• Score: 3.2625979527355784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic segmentation is essential for analysing anatomical features in biomedical research, yet a performance gap remains for Vision Transformers (ViTs) in the field, particularly for sparse, fine-structured, and low signal-to-noise targets. We attribute this challenge in part to the lightweight pixel decoders commonly used in promptable ViT models, who may lack the local inductive bias needed for high-precision biomedical masks. We bridge this gap by introducing ViTC-UNet, which conditions a UNet on frozen pre-trained ViT representations through learnable tokens and a two-way attention decoder. This combines ViT global visual priors with the local inductive bias and high-resolution decoding capacity of UNets, while avoiding end-to-end ViT fine-tuning even in cross-domain settings. ViTC-UNet outperforms baseline results in semantic segmentation tasks across MRI and CT modalities, demonstrating that structure-conditioned UNet decoding can efficiently adapt large-scale visual priors to high-complexity biomedical segmentation.

Related papers

• Rethinking the Adaptation of Vision Foundation Models for Efficient Cell Segmentation [38.881896184660164]
  EffiCell-Seg is a framework for highly efficient cell segmentation without re-training the visual encoder.<n>Our core insight is that pretrained VFMs intrinsically encode complementary structural priors.<n>EffiCell-Seg requires only 5M trainable parameters, over 130x fewer than fully fine-tuned VFM counterparts.
  arXiv  Detail & Related papers  (2026-06-20T07:15:14Z)
• Anatomy-Guided Representation Learning Using a Transformer-Based Network for Thyroid Nodule Segmentation in Ultrasound Images [16.78356926470714]
  We propose SSMT-Net, a Semi-Supervised Multi-Task Transformer-based Network that enhances Transformer-centric encoder feature extraction capability in an initial unsupervised phase.<n>In the supervised phase, the model jointly optimize nodule segmentation, gland segmentation, and size estimation, integrating both local and global contextual features.<n>In evaluations on the TN3K and DDTI datasets, SSMT-Net outperforms state-of-the-art methods, with higher accuracy and robustness, indicating its potential for real-world clinical applications.
  arXiv  Detail & Related papers  (2025-12-14T12:20:20Z)
• Boundary-Aware Vision Transformer for Angiography Vascular Network Segmentation [2.421170887647153]
  Boundary-Aware Vision Transformer (BAVT) is a ViT-based architecture enhanced with an edge-aware loss that explicitly guides the segmentation toward fine-grained vascular boundaries.<n>We validate our approach on the DCA-1 coronary angiography dataset, where BAVT achieves superior performance across medical image segmentation metrics.
  arXiv  Detail & Related papers  (2025-06-15T22:35:08Z)
• Your ViT is Secretly an Image Segmentation Model [50.71238842539735]
  Vision Transformers (ViTs) have shown remarkable performance and scalability across various computer vision tasks.<n>We show that inductive biases introduced by task-specific components can instead be learned by the ViT itself.<n>We introduce the Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct image segmentation.
  arXiv  Detail & Related papers  (2025-03-24T19:56:02Z)
• Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation [21.183229457060634]
  We pre-train Hi-End-MAE on a large-scale dataset of 10K CT scans and evaluate its performance across seven public medical image segmentation benchmarks.<n>Hi-End-MAE achieves superior transfer learning capabilities across various downstream tasks, revealing the potential of ViT in medical imaging applications.
  arXiv  Detail & Related papers  (2025-02-12T12:14:02Z)
• EViT-Unet: U-Net Like Efficient Vision Transformer for Medical Image Segmentation on Mobile and Edge Devices [5.307205032859535]
  We propose EViT-UNet, an efficient ViT-based segmentation network that reduces computational complexity while maintaining accuracy. EViT-UNet is built on a U-shaped architecture, comprising an encoder, decoder, bottleneck layer, and skip connections. Experimental results demonstrate that EViT-UNet achieves high accuracy in medical image segmentation while significantly reducing computational complexity.
  arXiv  Detail & Related papers  (2024-10-19T08:42:53Z)
• ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer [65.32312196621938]
  We propose Continual semantic benchmarks via Adapter-based ViT, namely ConSept. ConSept integrates lightweight attention-based adapters into vanilla ViTs. We propose two key strategies: distillation with a deterministic old-classes boundary for improved anti-catastrophic forgetting, and dual dice losses to regularize segmentation maps.
  arXiv  Detail & Related papers  (2024-02-26T15:51:45Z)
• Minimalist and High-Performance Semantic Segmentation with Plain Vision Transformers [10.72362704573323]
  We introduce the PlainSeg, a model comprising only three 3$times$3 convolutions in addition to the transformer layers. We also present the PlainSeg-Hier, which allows for the utilization of hierarchical features.
  arXiv  Detail & Related papers  (2023-10-19T14:01:40Z)
• DAT++: Spatially Dynamic Vision Transformer with Deformable Attention [87.41016963608067]
  We present Deformable Attention Transformer ( DAT++), a vision backbone efficient and effective for visual recognition. DAT++ achieves state-of-the-art results on various visual recognition benchmarks, with 85.9% ImageNet accuracy, 54.5 and 47.0 MS-COCO instance segmentation mAP, and 51.5 ADE20K semantic segmentation mIoU.
  arXiv  Detail & Related papers  (2023-09-04T08:26:47Z)
• SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers [76.13755422671822]
  This paper investigates the capability of plain Vision Transformers (ViTs) for semantic segmentation using the encoder-decoder framework. We introduce a novel Attention-to-Mask (atm) module to design a lightweight decoder effective for plain ViT. Our decoder outperforms the popular decoder UPerNet using various ViT backbones while consuming only about $5%$ of the computational cost.
  arXiv  Detail & Related papers  (2023-06-09T22:29:56Z)
• SegViT: Semantic Segmentation with Plain Vision Transformers [91.50075506561598]
  We explore the capability of plain Vision Transformers (ViTs) for semantic segmentation. We propose the Attention-to-Mask (ATM) module, in which similarity maps between a set of learnable class tokens and the spatial feature maps are transferred to the segmentation masks. Experiments show that our proposed SegVit using the ATM module outperforms its counterparts using the plain ViT backbone.
  arXiv  Detail & Related papers  (2022-10-12T00:30:26Z)
• Smoothing Matters: Momentum Transformer for Domain Adaptive Semantic Segmentation [48.7190017311309]
  We find that straightforwardly applying local ViTs in domain adaptive semantic segmentation does not bring in expected improvement. These high-frequency components make the training of local ViTs very unsmooth and hurt their transferability. In this paper, we introduce a low-pass filtering mechanism, momentum network, to smooth the learning dynamics of target domain features and pseudo labels.
  arXiv  Detail & Related papers  (2022-03-15T15:20:30Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.