SDTP: Semantic-aware Decoupled Transformer Pyramid for Dense Image Prediction

• URL: http://arxiv.org/abs/2109.08963v1
• Date: Sat, 18 Sep 2021 16:29:14 GMT
• Title: SDTP: Semantic-aware Decoupled Transformer Pyramid for Dense Image Prediction
• Authors: Zekun Li, Yufan Liu, Bing Li, Weiming Hu, Kebin Wu, Pei Wang
• Abstract summary: We propose a novel Semantic-aware Decoupled Transformer Pyramid (SDTP) for dense image prediction, consisting of Intra-level Semantic Promotion (ISP), Cross-level Decoupled Interaction (CDI) and Attention Refinement Function (ARF) ISP explores the semantic diversity in different receptive space. CDI builds the global attention and interaction among different levels in decoupled space which also solves the problem of heavy computation. Experimental results demonstrate the validity and generality of the proposed method, which outperforms the state-of-the-art by a significant margin in dense image prediction tasks.
• Score: 33.29925021875922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although transformer has achieved great progress on computer vision tasks, the scale variation in dense image prediction is still the key challenge. Few effective multi-scale techniques are applied in transformer and there are two main limitations in the current methods. On one hand, self-attention module in vanilla transformer fails to sufficiently exploit the diversity of semantic information because of its rigid mechanism. On the other hand, it is hard to build attention and interaction among different levels due to the heavy computational burden. To alleviate this problem, we first revisit multi-scale problem in dense prediction, verifying the significance of diverse semantic representation and multi-scale interaction, and exploring the adaptation of transformer to pyramidal structure. Inspired by these findings, we propose a novel Semantic-aware Decoupled Transformer Pyramid (SDTP) for dense image prediction, consisting of Intra-level Semantic Promotion (ISP), Cross-level Decoupled Interaction (CDI) and Attention Refinement Function (ARF). ISP explores the semantic diversity in different receptive space. CDI builds the global attention and interaction among different levels in decoupled space which also solves the problem of heavy computation. Besides, ARF is further added to refine the attention in transformer. Experimental results demonstrate the validity and generality of the proposed method, which outperforms the state-of-the-art by a significant margin in dense image prediction tasks. Furthermore, the proposed components are all plug-and-play, which can be embedded in other methods.

Related papers

• DAPE V2: Process Attention Score as Feature Map for Length Extrapolation [63.87956583202729]
  We conceptualize attention as a feature map and apply the convolution operator to mimic the processing methods in computer vision. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution.
  arXiv  Detail & Related papers  (2024-10-07T07:21:49Z)
• CT-MVSNet: Efficient Multi-View Stereo with Cross-scale Transformer [8.962657021133925]
  Cross-scale transformer (CT) processes feature representations at different stages without additional computation. We introduce an adaptive matching-aware transformer (AMT) that employs different interactive attention combinations at multiple scales. We also present a dual-feature guided aggregation (DFGA) that embeds the coarse global semantic information into the finer cost volume construction.
  arXiv  Detail & Related papers  (2023-12-14T01:33:18Z)
• Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images [1.662438436885552]
  Multi-modal fusion has been determined to enhance the accuracy by fusing data from multiple modalities. We propose a novel multi-modal fusion strategy for mapping relationships between different channels at the early stage. By addressing fusion in the early stage, as opposed to mid or late-stage methods, our method achieves competitive and even superior performance compared to existing techniques.
  arXiv  Detail & Related papers  (2023-10-21T00:56:11Z)
• Transformer-based Multimodal Change Detection with Multitask Consistency Constraints [10.906283981247796]
  Current change detection methods struggle with the multitask conflicts between semantic and height change detection tasks. We propose an efficient Transformer-based network that learns shared representation between cross-dimensional inputs through cross-attention. Compared to five state-of-the-art change detection methods, our model demonstrates consistent multitask superiority in terms of semantic and height change detection.
  arXiv  Detail & Related papers  (2023-10-13T17:38:45Z)
• Inverted Pyramid Multi-task Transformer for Dense Scene Understanding [11.608682595506354]
  We propose a novel end-to-end Inverted Pyramid multi-task Transformer (InvPT) to perform simultaneous modeling of spatial positions and multiple tasks in a unified framework. InvPT presents an efficient UP-Transformer block to learn multi-task feature interaction at gradually increased resolutions. Our method achieves superior multi-task performance on NYUD-v2 and PASCAL-Context datasets respectively, and significantly outperforms previous state-of-the-arts.
  arXiv  Detail & Related papers  (2022-03-15T15:29:08Z)
• XAI for Transformers: Better Explanations through Conservative Propagation [60.67748036747221]
  We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. Our proposal can be seen as a proper extension of the well-established LRP method to Transformers.
  arXiv  Detail & Related papers  (2022-02-15T10:47:11Z)
• Blending Anti-Aliasing into Vision Transformer [57.88274087198552]
  discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguishable distortions. We propose a plug-and-play Aliasing-Reduction Module(ARM) to alleviate the aforementioned issue.
  arXiv  Detail & Related papers  (2021-10-28T14:30:02Z)
• DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation [18.755217252996754]
  We propose a novel deep medical image segmentation framework called Dual Swin Transformer U-Net (DS-TransUNet) Unlike many prior Transformer-based solutions, the proposed DS-TransUNet first adopts dual-scale encoderworks based on Swin Transformer to extract the coarse and fine-grained feature representations of different semantic scales. As the core component for our DS-TransUNet, a well-designed Transformer Interactive Fusion (TIF) module is proposed to effectively establish global dependencies between features of different scales through the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-06-12T08:37:17Z)
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias [76.16156833138038]
  We propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context. In each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network.
  arXiv  Detail & Related papers  (2021-06-07T05:31:06Z)
• Less is More: Pay Less Attention in Vision Transformers [61.05787583247392]
  Less attention vIsion Transformer builds upon the fact that convolutions, fully-connected layers, and self-attentions have almost equivalent mathematical expressions for processing image patch sequences. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation.
  arXiv  Detail & Related papers  (2021-05-29T05:26:07Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.