Abstract: Medical image segmentation is an essential prerequisite for developing
healthcare systems, especially for disease diagnosis and treatment planning. On
various medical image segmentation tasks, the u-shaped architecture, also known
as U-Net, has become the de-facto standard and achieved tremendous success.
However, due to the intrinsic locality of convolution operations, U-Net
generally demonstrates limitations in explicitly modeling long-range
dependency. Transformers, designed for sequence-to-sequence prediction, have
emerged as alternative architectures with innate global self-attention
mechanisms, but can result in limited localization abilities due to
insufficient low-level details. In this paper, we propose TransUNet, which
merits both Transformers and U-Net, as a strong alternative for medical image
segmentation. On one hand, the Transformer encodes tokenized image patches from
a convolution neural network (CNN) feature map as the input sequence for
extracting global contexts. On the other hand, the decoder upsamples the
encoded features which are then combined with the high-resolution CNN feature
maps to enable precise localization.
We argue that Transformers can serve as strong encoders for medical image
segmentation tasks, with the combination of U-Net to enhance finer details by
recovering localized spatial information. TransUNet achieves superior
performances to various competing methods on different medical applications
including multi-organ segmentation and cardiac segmentation. Code and models
are available at https://github.com/Beckschen/T ransUNet.