MultiScale Spectral-Spatial Convolutional Transformer for Hyperspectral Image Classification

• URL: http://arxiv.org/abs/2310.18550v1
• Date: Sat, 28 Oct 2023 00:41:35 GMT
• Title: MultiScale Spectral-Spatial Convolutional Transformer for Hyperspectral Image Classification
• Authors: Zhiqiang Gong, Xian Zhou, Wen Yao
• Abstract summary: Transformer has become an alternative architecture of CNNs for hyperspectral image classification. We propose a multiscale spectral-spatial convolutional Transformer (MultiscaleFormer) for hyperspectral image classification.
• Score: 9.051982753583232
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Due to the powerful ability in capturing the global information, Transformer has become an alternative architecture of CNNs for hyperspectral image classification. However, general Transformer mainly considers the global spectral information while ignores the multiscale spatial information of the hyperspectral image. In this paper, we propose a multiscale spectral-spatial convolutional Transformer (MultiscaleFormer) for hyperspectral image classification. First, the developed method utilizes multiscale spatial patches as tokens to formulate the spatial Transformer and generates multiscale spatial representation of each band in each pixel. Second, the spatial representation of all the bands in a given pixel are utilized as tokens to formulate the spectral Transformer and generate the multiscale spectral-spatial representation of each pixel. Besides, a modified spectral-spatial CAF module is constructed in the MultiFormer to fuse cross-layer spectral and spatial information. Therefore, the proposed MultiFormer can capture the multiscale spectral-spatial information and provide better performance than most of other architectures for hyperspectral image classification. Experiments are conducted over commonly used real-world datasets and the comparison results show the superiority of the proposed method.

Related papers

• Hierarchical Homogeneity-Based Superpixel Segmentation: Application to Hyperspectral Image Analysis [11.612069983959985]
  We propose a multiscale superpixel method that is computationally efficient for processing hyperspectral data. The proposed hierarchical approach leads to superpixels of variable sizes but with higher spectral homogeneity. For validation, the proposed homogeneity-based hierarchical method was applied as a preprocessing step in the spectral unmixing and classification tasks.
  arXiv  Detail & Related papers  (2024-07-22T01:20:32Z)
• HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model [88.13261547704444]
  Hyper SIGMA is a vision transformer-based foundation model for HSI interpretation. It integrates spatial and spectral features using a specially designed spectral enhancement module. It shows significant advantages in scalability, robustness, cross-modal transferring capability, and real-world applicability.
  arXiv  Detail & Related papers  (2024-06-17T13:22:58Z)
• TOP-ReID: Multi-spectral Object Re-Identification with Token Permutation [64.65950381870742]
  We propose a cyclic token permutation framework for multi-spectral object ReID, dubbled TOP-ReID. We also propose a Token Permutation Module (TPM) for cyclic multi-spectral feature aggregation. Our proposed framework can generate more discriminative multi-spectral features for robust object ReID.
  arXiv  Detail & Related papers  (2023-12-15T08:54:15Z)
• Cross-Scope Spatial-Spectral Information Aggregation for Hyperspectral Image Super-Resolution [47.12985199570964]
  We propose a novel cross-scope spatial-spectral Transformer (CST) to investigate long-range spatial and spectral similarities for single hyperspectral image super-resolution. Specifically, we devise cross-attention mechanisms in spatial and spectral dimensions to comprehensively model the long-range spatial-spectral characteristics. Experiments over three hyperspectral datasets demonstrate that the proposed CST is superior to other state-of-the-art methods both quantitatively and visually.
  arXiv  Detail & Related papers  (2023-11-29T03:38:56Z)
• Multiview Transformer: Rethinking Spatial Information in Hyperspectral Image Classification [43.17196501332728]
  Identifying the land cover category for each pixel in a hyperspectral image relies on spectral and spatial information. In this article, we investigate that scene-specific but not essential correlations may be recorded in an HSI cuboid. We propose a multiview transformer for HSI classification, which consists of multiview principal component analysis (MPCA), spectral encoder-decoder (SED), and spatial-pooling tokenization transformer (SPTT)
  arXiv  Detail & Related papers  (2023-10-11T04:25:24Z)
• Multi-Scale U-Shape MLP for Hyperspectral Image Classification [13.85573689689951]
  Two challenges in identifying pixels of the hyperspectral image are respectively representing the correlated information among the local and global, as well as the abundant parameters of the model. We propose a Multi-Scale U-shape Multi-Layer Perceptron (MUMLP) model consisting of the designed MSC (Multi-Scale Channel) block and the U-shape Multi-Layer Perceptron structure. Our model can outperform state-of-the-art methods across-the-board on three wide-adopted public datasets.
  arXiv  Detail & Related papers  (2023-07-05T08:52:27Z)
• AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation [7.415370401064414]
  We propose AerialFormer, which unifies Transformers at the contracting path with lightweight Multi-Dilated Conal Neural Networks (MD-CNNs) at the expanding path. Our AerialFormer is designed as a hierarchical structure, in which Transformer outputs multi-scale features and MD-CNNs decoder aggregates information from the multi-scales. We have benchmarked AerialFormer on three common datasets including iSAID, LoveDA, and Potsdam.
  arXiv  Detail & Related papers  (2023-06-12T03:28:18Z)
• Probabilistic Deep Metric Learning for Hyperspectral Image Classification [91.5747859691553]
  This paper proposes a probabilistic deep metric learning framework for hyperspectral image classification. It aims to predict the category of each pixel for an image captured by hyperspectral sensors. Our framework can be readily applied to existing hyperspectral image classification methods.
  arXiv  Detail & Related papers  (2022-11-15T17:57:12Z)
• HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening [60.89777029184023]
  Pansharpening aims to fuse a registered high-resolution panchromatic image (PAN) with a low-resolution hyperspectral image (LR-HSI) to generate an enhanced HSI with high spectral and spatial resolution. Existing pansharpening approaches neglect using an attention mechanism to transfer HR texture features from PAN to LR-HSI features, resulting in spatial and spectral distortions. We present a novel attention mechanism for pansharpening called HyperTransformer, in which features of LR-HSI and PAN are formulated as queries and keys in a transformer, respectively.
  arXiv  Detail & Related papers  (2022-03-04T18:59:08Z)
• SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers [91.09957836250209]
  Hyperspectral (HS) images are characterized by approximately contiguous spectral information. CNNs have been proven to be a powerful feature extractor in HS image classification. We propose a novel backbone network called ulSpectralFormer for HS image classification.
  arXiv  Detail & Related papers  (2021-07-07T02:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.