Improved Image Classification with Token Fusion

• URL: http://arxiv.org/abs/2208.09183v1
• Date: Fri, 19 Aug 2022 07:02:50 GMT
• Title: Improved Image Classification with Token Fusion
• Abstract summary: We propose a method using the fusion of CNN and transformer structure to improve image classification performance. In an experiment using ImageNet 1k, the proposed method shows the best classification performance.
• Score: 9.413225906164838
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we propose a method using the fusion of CNN and transformer structure to improve image classification performance. In the case of CNN, information about a local area on an image can be extracted well, but there is a limit to the extraction of global information. On the other hand, the transformer has an advantage in relatively global extraction, but has a disadvantage in that it requires a lot of memory for local feature value extraction. In the case of an image, it is converted into a feature map through CNN, and each feature map's pixel is considered a token. At the same time, the image is divided into patch areas and then fused with the transformer method that views them as tokens. For the fusion of tokens with two different characteristics, we propose three methods: (1) late token fusion with parallel structure, (2) early token fusion, (3) token fusion in a layer by layer. In an experiment using ImageNet 1k, the proposed method shows the best classification performance.

Related papers

• Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era [60.09990228573728]
  We introduce some learnable aggregation tokens, which are prepended to the patch tokens before a particular transformer block.<n>All these tokens will be jointly processed and interact globally via the intrinsic self-attention mechanism.<n>Our method outperforms state-of-the-art methods on several VPR datasets with higher efficiency and ranks 1st on the MSLS challenge leaderboard.
  arXiv  Detail & Related papers  (2025-11-08T14:35:11Z)
• TCFormer: Visual Recognition via Token Clustering Transformer [79.24723479088097]
  We propose the Token Clustering Transformer (TCFormer), which generates dynamic vision tokens based on semantic meaning. Our dynamic tokens possess two crucial characteristics: (1) Representing image regions with similar semantic meanings using the same vision token, even if those regions are not adjacent, and (2) concentrating on regions with valuable details and represent them using fine tokens.
  arXiv  Detail & Related papers  (2024-07-16T02:26:18Z)
• Transformer based Pluralistic Image Completion with Reduced Information Loss [72.92754600354199]
  Transformer based methods have achieved great success in image inpainting recently. They regard each pixel as a token, thus suffering from an information loss issue. We propose a new transformer based framework called "PUT"
  arXiv  Detail & Related papers  (2024-03-31T01:20:16Z)
• Fusion Transformer with Object Mask Guidance for Image Forgery Analysis [9.468075384561947]
  We introduce OMG-Fuser, a fusion transformer-based network designed to extract information from various forensic signals. Our approach can operate with an arbitrary number of forensic signals and leverages object information for their analysis. Our model is robust against traditional and novel forgery attacks and can be expanded with new signals without training from scratch.
  arXiv  Detail & Related papers  (2024-03-18T20:20:13Z)
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging [71.84591084401458]
  Vision Transformers (ViTs) have emerged as powerful backbones in computer vision, outperforming many traditional CNNs. computational overhead, largely attributed to the self-attention mechanism, makes deployment on resource-constrained edge devices challenging. We introduce "Token Fusion" (ToFu), a method that amalgamates the benefits of both token pruning and token merging.
  arXiv  Detail & Related papers  (2023-12-02T04:29:19Z)
• TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization [81.70547404891099]
  CNN-based methods for cross-view image geo-localization fail to model global correlation. We propose a pure transformer-based approach (TransGeo) to address these limitations. TransGeo achieves state-of-the-art results on both urban and rural datasets.
  arXiv  Detail & Related papers  (2022-03-31T21:19:41Z)
• Learning Tracking Representations via Dual-Branch Fully Transformer Networks [82.21771581817937]
  We present a Siamese-like Dual-branch network based on solely Transformers for tracking. We extract a feature vector for each patch based on its matching results with others within an attention window. The method achieves better or comparable results as the best-performing methods.
  arXiv  Detail & Related papers  (2021-12-05T13:44:33Z)
• Semantic Image Fusion [2.4366811507669124]
  This paper proposes a novel system for the semantic combination of visual content using pre-trained CNN network architectures. Simple "choose maximum" and "local majority" filter based fusion rules are utilised for feature map fusion. The developed methods are able to give equivalent low-level fusion performance to state of the art methods.
  arXiv  Detail & Related papers  (2021-10-13T13:15:16Z)
• Image Fusion Transformer [75.71025138448287]
  In image fusion, images obtained from different sensors are fused to generate a single image with enhanced information. In recent years, state-of-the-art methods have adopted Convolution Neural Networks (CNNs) to encode meaningful features for image fusion. We propose a novel Image Fusion Transformer (IFT) where we develop a transformer-based multi-scale fusion strategy.
  arXiv  Detail & Related papers  (2021-07-19T16:42:49Z)
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification [17.709880544501758]
  We propose a dual-branch transformer to combine image patches of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise.
  arXiv  Detail & Related papers  (2021-03-27T13:03:17Z)
• NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models [12.16870022547833]
  We propose a novel method for infrared and visible image fusion. We develop nest connection-based network and spatial/channel attention models. Experiments are performed on publicly available datasets.
  arXiv  Detail & Related papers  (2020-07-01T08:46:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.