Transformers Meet Hyperspectral Imaging: A Comprehensive Study of Models, Challenges and Open Problems

• URL: http://arxiv.org/abs/2506.08596v1
• Date: Tue, 10 Jun 2025 09:04:30 GMT
• Title: Transformers Meet Hyperspectral Imaging: A Comprehensive Study of Models, Challenges and Open Problems
• Authors: Guyang Zhang, Waleed Abdulla,
• Abstract summary: We review more than 300 papers published up to 2025 and present the first end-to-end survey dedicated to Transformer-based HSI classification.<n>The study categorizes every stage of a typical pipeline-pre-processing, patch or pixel tokenization, positional encoding, spatial-spectral feature extraction, multi-head self-attention variants, skip connections, and loss design.<n>We outline a research agenda prioritizing valuable public data sets, lightweight on-edge models, illumination and sensor shifts, and intrinsically interpretable attention mechanisms.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers have become the architecture of choice for learning long-range dependencies, yet their adoption in hyperspectral imaging (HSI) is still emerging. We reviewed more than 300 papers published up to 2025 and present the first end-to-end survey dedicated to Transformer-based HSI classification. The study categorizes every stage of a typical pipeline-pre-processing, patch or pixel tokenization, positional encoding, spatial-spectral feature extraction, multi-head self-attention variants, skip connections, and loss design-and contrasts alternative design choices with the unique spatial-spectral properties of HSI. We map the field's progress against persistent obstacles: scarce labeled data, extreme spectral dimensionality, computational overhead, and limited model explainability. Finally, we outline a research agenda prioritizing valuable public data sets, lightweight on-edge models, illumination and sensor shifts robustness, and intrinsically interpretable attention mechanisms. Our goal is to guide researchers in selecting, combining, or extending Transformer components that are truly fit for purpose for next-generation HSI applications.

Related papers

• Dual Selective Fusion Transformer Network for Hyperspectral Image Classification [34.7051033596479]
  Transformer has achieved satisfactory results in the field of hyperspectral image (HSI) classification.<n>Existing Transformer models face two key challenges when dealing with HSI scenes characterized by diverse land cover types and rich spectral information.<n>We propose a novel Dual Selective Fusion Transformer Network (DSFormer) for HSI classification.
  arXiv  Detail & Related papers  (2024-10-04T06:05:26Z)
• Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increase [2.6513322539118582]
  We explore the challenges and evolution of two key technologies in the current field of AI: Vision Transformer model and Large Language Model (LLM) Vision Transformer captures global information by splitting images into small pieces, but its high reference count and compute overhead limit deployment on mobile devices. LLM has revolutionized natural language processing, but it also faces huge deployment challenges.
  arXiv  Detail & Related papers  (2024-08-16T11:56:49Z)
• Geometric Features Enhanced Human-Object Interaction Detection [11.513009304308724]
  We propose a novel end-to-end Transformer-style HOI detection model, i.e., geometric features enhanced HOI detector (GeoHOI) One key part of the model is a new unified self-supervised keypoint learning method named UniPointNet. GeoHOI effectively upgrades a Transformer-based HOI detector benefiting from the keypoints similarities measuring the likelihood of human-object interactions.
  arXiv  Detail & Related papers  (2024-06-26T18:52:53Z)
• S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR [50.435592120607815]
  Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR) Previous works have primarily relied on multi-stage learning, where the generated semantic scene graphs depend on intermediate processes with pose estimation and object detection. In this study, we introduce a novel single-stage bi-modal transformer framework for SGG in the OR, termed S2Former-OR.
  arXiv  Detail & Related papers  (2024-02-22T11:40:49Z)
• GenFace: A Large-Scale Fine-Grained Face Forgery Benchmark and Cross Appearance-Edge Learning [50.7702397913573]
  The rapid advancement of photorealistic generators has reached a critical juncture where the discrepancy between authentic and manipulated images is increasingly indistinguishable. Although there have been a number of publicly available face forgery datasets, the forgery faces are mostly generated using GAN-based synthesis technology. We propose a large-scale, diverse, and fine-grained high-fidelity dataset, namely GenFace, to facilitate the advancement of deepfake detection.
  arXiv  Detail & Related papers  (2024-02-03T03:13:50Z)
• FactoFormer: Factorized Hyperspectral Transformers with Self-Supervised Pretraining [36.44039681893334]
  Hyperspectral images (HSIs) contain rich spectral and spatial information. Current state-of-the-art hyperspectral transformers only tokenize the input HSI sample along the spectral dimension. We propose a novel factorized spectral-spatial transformer that incorporates factorized self-supervised pretraining procedures.
  arXiv  Detail & Related papers  (2023-09-18T02:05:52Z)
• ViT-Calibrator: Decision Stream Calibration for Vision Transformer [49.60474757318486]
  We propose a new paradigm dubbed Decision Stream that boosts the performance of general Vision Transformers. We shed light on the information propagation mechanism in the learning procedure by exploring the correlation between different tokens and the relevance coefficient of multiple dimensions.
  arXiv  Detail & Related papers  (2023-04-10T02:40:24Z)
• ViTs for SITS: Vision Transformers for Satellite Image Time Series [52.012084080257544]
  We introduce a fully-attentional model for general Satellite Image Time Series (SITS) processing based on the Vision Transformer (ViT) TSViT splits a SITS record into non-overlapping patches in space and time which are tokenized and subsequently processed by a factorized temporo-spatial encoder.
  arXiv  Detail & Related papers  (2023-01-12T11:33:07Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.