SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness
for Hyperspectral Object Tracking
- URL: http://arxiv.org/abs/2403.05852v1
- Date: Sat, 9 Mar 2024 09:37:13 GMT
- Title: SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness
for Hyperspectral Object Tracking
- Authors: Hanzheng Wang, Wei Li, Xiang-Gen Xia, Qian Du, and Jing Tian
- Abstract summary: Hyperspectral video (HSV) offers valuable spatial, spectral, and temporal information simultaneously.
Existing methods primarily focus on band regrouping and rely on RGB trackers for feature extraction.
In this paper, a spatial-spectral fusion network with spectral angle awareness (SST-Net) is proposed for hyperspectral (HS) object tracking.
- Score: 21.664141982246598
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Hyperspectral video (HSV) offers valuable spatial, spectral, and temporal
information simultaneously, making it highly suitable for handling challenges
such as background clutter and visual similarity in object tracking. However,
existing methods primarily focus on band regrouping and rely on RGB trackers
for feature extraction, resulting in limited exploration of spectral
information and difficulties in achieving complementary representations of
object features. In this paper, a spatial-spectral fusion network with spectral
angle awareness (SST-Net) is proposed for hyperspectral (HS) object tracking.
Firstly, to address the issue of insufficient spectral feature extraction in
existing networks, a spatial-spectral feature backbone ($S^2$FB) is designed.
With the spatial and spectral extraction branch, a joint representation of
texture and spectrum is obtained. Secondly, a spectral attention fusion module
(SAFM) is presented to capture the intra- and inter-modality correlation to
obtain the fused features from the HS and RGB modalities. It can incorporate
the visual information into the HS spectral context to form a robust
representation. Thirdly, to ensure a more accurate response of the tracker to
the object position, a spectral angle awareness module (SAAM) investigates the
region-level spectral similarity between the template and search images during
the prediction stage. Furthermore, we develop a novel spectral angle awareness
loss (SAAL) to offer guidance for the SAAM based on similar regions. Finally,
to obtain the robust tracking results, a weighted prediction method is
considered to combine the HS and RGB predicted motions of objects to leverage
the strengths of each modality. Extensive experiments on the HOTC dataset
demonstrate the effectiveness of the proposed SSF-Net, compared with
state-of-the-art trackers.
Related papers
- HTD-Mamba: Efficient Hyperspectral Target Detection with Pyramid State Space Model [5.505983410956103]
Hyperspectral target detection (HTD) identifies objects of interest from complex backgrounds at the pixel level.
This paper proposes an efficient self-supervised HTD method with a pyramid state space model (SSM), named HTD-Mamba.
Experiments conducted on four public datasets demonstrate that the proposed method outperforms state-of-the-art methods in both quantitative and qualitative evaluations.
arXiv Detail & Related papers (2024-07-09T13:21:26Z) - HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model [88.13261547704444]
Hyper SIGMA is a vision transformer-based foundation model for HSI interpretation.
It integrates spatial and spectral features using a specially designed spectral enhancement module.
It shows significant advantages in scalability, robustness, cross-modal transferring capability, and real-world applicability.
arXiv Detail & Related papers (2024-06-17T13:22:58Z) - Hyperspectral Image Reconstruction via Combinatorial Embedding of
Cross-Channel Spatio-Spectral Clues [6.580484964018551]
Existing learning-based hyperspectral reconstruction methods show limitations in fully exploiting the information among the hyperspectral bands.
We propose to investigate the inter-dependencies in their respective hyperspectral space.
These embedded features can be fully exploited by querying the inter-channel correlations.
arXiv Detail & Related papers (2023-12-18T11:37:19Z) - Spectrum-driven Mixed-frequency Network for Hyperspectral Salient Object
Detection [14.621504062838731]
We propose a novel approach that fully leverages the spectral characteristics by extracting two distinct frequency components from the spectrum.
The Spectral Saliency approximates the region of salient objects, while the Spectral Edge captures edge information of salient objects.
To effectively utilize this dual-frequency information, we introduce a novel lightweight Spectrum-driven Mixed-frequency Network (SMN)
arXiv Detail & Related papers (2023-12-02T08:05:45Z) - Cross-Scope Spatial-Spectral Information Aggregation for Hyperspectral
Image Super-Resolution [47.12985199570964]
We propose a novel cross-scope spatial-spectral Transformer (CST) to investigate long-range spatial and spectral similarities for single hyperspectral image super-resolution.
Specifically, we devise cross-attention mechanisms in spatial and spectral dimensions to comprehensively model the long-range spatial-spectral characteristics.
Experiments over three hyperspectral datasets demonstrate that the proposed CST is superior to other state-of-the-art methods both quantitatively and visually.
arXiv Detail & Related papers (2023-11-29T03:38:56Z) - ESSAformer: Efficient Transformer for Hyperspectral Image
Super-resolution [76.7408734079706]
Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation.
We propose ESSAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure.
arXiv Detail & Related papers (2023-07-26T07:45:14Z) - Object Detection in Hyperspectral Image via Unified Spectral-Spatial
Feature Aggregation [55.9217962930169]
We present S2ADet, an object detector that harnesses the rich spectral and spatial complementary information inherent in hyperspectral images.
S2ADet surpasses existing state-of-the-art methods, achieving robust and reliable results.
arXiv Detail & Related papers (2023-06-14T09:01:50Z) - Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image
Reconstruction [127.20208645280438]
Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement.
Modeling the inter-spectra interactions is beneficial for HSI reconstruction.
Mask-guided Spectral-wise Transformer (MST) proposes a novel framework for HSI reconstruction.
arXiv Detail & Related papers (2021-11-15T16:59:48Z) - Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral
Imagery [79.69449412334188]
In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches.
We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data.
Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
arXiv Detail & Related papers (2020-05-18T14:25:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.