Transformer in Transformer
- URL: http://arxiv.org/abs/2103.00112v1
- Date: Sat, 27 Feb 2021 03:12:16 GMT
- Title: Transformer in Transformer
- Authors: Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, Yunhe Wang
- Abstract summary: We propose a novel Transformer-iN-Transformer (TNT) model for modeling both patch-level and pixel-level representation.
Our TNT achieves $81.3%$ top-1 accuracy on ImageNet which is $1.5%$ higher than that of DeiT with similar computational cost.
- Score: 59.066686278998354
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Transformer is a type of self-attention-based neural networks originally
applied for NLP tasks. Recently, pure transformer-based models are proposed to
solve computer vision problems. These visual transformers usually view an image
as a sequence of patches while they ignore the intrinsic structure information
inside each patch. In this paper, we propose a novel Transformer-iN-Transformer
(TNT) model for modeling both patch-level and pixel-level representation. In
each TNT block, an outer transformer block is utilized to process patch
embeddings, and an inner transformer block extracts local features from pixel
embeddings. The pixel-level feature is projected to the space of patch
embedding by a linear transformation layer and then added into the patch. By
stacking the TNT blocks, we build the TNT model for image recognition.
Experiments on ImageNet benchmark and downstream tasks demonstrate the
superiority and efficiency of the proposed TNT architecture. For example, our
TNT achieves $81.3\%$ top-1 accuracy on ImageNet which is $1.5\%$ higher than
that of DeiT with similar computational cost. The code will be available at
https://github.com/huawei-noah/noah-research/tree/master/TNT.
Related papers
- Patch Is Not All You Need [57.290256181083016]
We propose a novel Pattern Transformer to adaptively convert images to pattern sequences for Transformer input.
We employ the Convolutional Neural Network to extract various patterns from the input image.
We have accomplished state-of-the-art performance on CIFAR-10 and CIFAR-100, and have achieved competitive results on ImageNet.
arXiv Detail & Related papers (2023-08-21T13:54:00Z) - Pix4Point: Image Pretrained Standard Transformers for 3D Point Cloud
Understanding [62.502694656615496]
We present Progressive Point Patch Embedding and present a new point cloud Transformer model namely PViT.
PViT shares the same backbone as Transformer but is shown to be less hungry for data, enabling Transformer to achieve performance comparable to the state-of-the-art.
We formulate a simple yet effective pipeline dubbed "Pix4Point" that allows harnessing Transformers pretrained in the image domain to enhance downstream point cloud understanding.
arXiv Detail & Related papers (2022-08-25T17:59:29Z) - PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid
Architecture [46.252298619903165]
Transformer-in-Transformer (TNT) architecture utilizes inner transformer and outer transformer to extract both local and global representations.
New "PyramidTNT" significantly improves the original TNT by establishing hierarchical representations.
PyramidTNT achieves better performances than the previous state-of-the-art vision transformers such as Swin Transformer.
arXiv Detail & Related papers (2022-01-04T04:56:57Z) - Vision Transformer with Progressive Sampling [73.60630716500154]
We propose an iterative and progressive sampling strategy to locate discriminative regions.
When trained from scratch on ImageNet, PS-ViT performs 3.8% higher than the vanilla ViT in terms of top-1 accuracy.
arXiv Detail & Related papers (2021-08-03T18:04:31Z) - PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion [37.993611194758195]
We propose a Patch PyramidTransformer(PPT) to address the issues of extracting semantic information from an image.
The experimental results demonstrate its superior performance against the state-of-the-art fusion approaches.
arXiv Detail & Related papers (2021-07-29T13:57:45Z) - Fully Transformer Networks for Semantic ImageSegmentation [26.037770622551882]
We explore a novel framework for semantic image segmentation, which is encoder-decoder based Fully Transformer Networks (FTN)
We propose a Pyramid Group Transformer (PGT) as the encoder for progressively learning hierarchical features, while reducing the computation complexity of the standard visual transformer(ViT)
Then, we propose a Feature Pyramid Transformer (FPT) to fuse semantic-level and spatial-level information from multiple levels of the PGT encoder for semantic image segmentation.
arXiv Detail & Related papers (2021-06-08T05:15:28Z) - Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation [63.46694853953092]
Swin-Unet is an Unet-like pure Transformer for medical image segmentation.
tokenized image patches are fed into the Transformer-based U-shaped decoder-Decoder architecture.
arXiv Detail & Related papers (2021-05-12T09:30:26Z) - Incorporating Convolution Designs into Visual Transformers [24.562955955312187]
We propose a new textbfConvolution-enhanced image Transformer (CeiT) which combines the advantages of CNNs in extracting low-level features, strengthening locality, and the advantages of Transformers in establishing long-range dependencies.
Experimental results on ImageNet and seven downstream tasks show the effectiveness and generalization ability of CeiT compared with previous Transformers and state-of-the-art CNNs, without requiring a large amount of training data and extra CNN teachers.
arXiv Detail & Related papers (2021-03-22T13:16:12Z) - Transformer for Image Quality Assessment [14.975436239088312]
We propose an architecture of using a shallow Transformer encoder on the top of a feature map extracted by convolution neural networks (CNN)
Adaptive positional embedding is employed in the Transformer encoder to handle images with arbitrary resolutions.
We have found that the proposed TRIQ architecture achieves outstanding performance.
arXiv Detail & Related papers (2020-12-30T18:43:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.