SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping

• URL: http://arxiv.org/abs/2510.27219v1
• Date: Fri, 31 Oct 2025 06:28:14 GMT
• Title: SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping
• Authors: Renjie Ji, Xue Wang, Chao Niu, Wen Zhang, Yong Mei, Kun Tan,
• Abstract summary: SpecAware is a novel hyperspectral spectral-content aware foundation model for unifying multi-sensor learning for HSI mapping.<n>The core of SpecAware is a two-step hypernetwork-driven encoding process for HSI data.<n>Experiments on six datasets demonstrate that SpecAware can learn superior feature representations.
• Score: 9.392313789126714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperspectral imaging (HSI) is a vital tool for fine-grained land-use and land-cover (LULC) mapping. However, the inherent heterogeneity of HSI data has long posed a major barrier to developing generalized models via joint training. Although HSI foundation models have shown promise for different downstream tasks, the existing approaches typically overlook the critical guiding role of sensor meta-attributes, and struggle with multi-sensor training, limiting their transferability. To address these challenges, we propose SpecAware, which is a novel hyperspectral spectral-content aware foundation model for unifying multi-sensor learning for HSI mapping. We also constructed the Hyper-400K dataset to facilitate this research, which is a new large-scale, high-quality benchmark dataset with over 400k image patches from diverse airborne AVIRIS sensors. The core of SpecAware is a two-step hypernetwork-driven encoding process for HSI data. Firstly, we designed a meta-content aware module to generate a unique conditional input for each HSI patch, tailored to each spectral band of every sample by fusing the sensor meta-attributes and its own image content. Secondly, we designed the HyperEmbedding module, where a sample-conditioned hypernetwork dynamically generates a pair of matrix factors for channel-wise encoding, consisting of adaptive spatial pattern extraction and latent semantic feature re-projection. Thus, SpecAware gains the ability to perceive and interpret spatial-spectral features across diverse scenes and sensors. This, in turn, allows SpecAware to adaptively process a variable number of spectral channels, establishing a unified framework for joint pre-training. Extensive experiments on six datasets demonstrate that SpecAware can learn superior feature representations, excelling in land-cover semantic segmentation classification, change detection, and scene classification.

Related papers

• The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding [82.53463660564933]
  semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders retain high-frequency information that conveys fine-grained detail.<n>We propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator.
  arXiv  Detail & Related papers  (2025-12-22T18:59:57Z)
• Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models [18.24287471339871]
  Hyperspectral imaging (HSI) captures spatial information along with dense spectral measurements across numerous narrow wavelength bands.<n>Our architecture incorporates a spectral transformer and a spectrum-aware spatial prior module to extract rich spatial-spectral features.<n>Our architecture achieves state-of-the-art semantic segmentation performance while directly using HSI inputs, outperforming both vision-based and hyperspectral segmentation methods.
  arXiv  Detail & Related papers  (2025-09-24T13:32:07Z)
• Hyperspectral Mamba for Hyperspectral Object Tracking [56.365517163296936]
  A new hyperspectral object tracking network equipped with Mamba (HyMamba) is proposed.<n>It unifies spectral, cross-depth, and temporal modeling through state space modules (SSMs)<n>HyMamba achieves state-of-the-art performance on seven benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-10T03:47:43Z)
• GCRPNet: Graph-Enhanced Contextual and Regional Perception Network for Salient Object Detection in Optical Remote Sensing Images [68.33481681452675]
  We propose a graph-enhanced contextual and regional perception network (GCRPNet)<n>It builds upon the Mamba architecture to simultaneously capture long-range dependencies and enhance regional feature representation.<n>It performs adaptive patch scanning on feature maps processed via multi-scale convolutions, thereby capturing rich local region information.
  arXiv  Detail & Related papers  (2025-08-14T11:31:43Z)
• SpectralX: Parameter-efficient Domain Generalization for Spectral Remote Sensing Foundation Models [27.717648142854696]
  SpectralX is an innovative parameter-efficient fine-tuning framework that adapt existing RSFMs as backbone.<n>We develop an Attribute-oriented Mixture of Adapter (AoMoA) that dynamically aggregates multi-attribute expert knowledge.<n>By iteratively querying low-level semantic features with high-level representations, the model learns to focus on task-beneficial attributes.
  arXiv  Detail & Related papers  (2025-08-03T12:14:38Z)
• AuxDet: Auxiliary Metadata Matters for Omni-Domain Infrared Small Target Detection [49.81255045696323]
  We present the Auxiliary Metadata Driven Infrared Small Target Detector (AuxDet)<n>AuxDet integrates metadata semantics with visual features, guiding adaptive representation learning for each sample.<n>Experiments on the challenging WideIRSTD-Full benchmark demonstrate that AuxDet consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-21T07:02:05Z)
• Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction [20.1863553357121]
  Current deep learning architectures for remote sensing are fundamentally rigid.<n>We introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling.<n> STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands.<n>It unifies various dense prediction tasks and diverse semantic class predictions.
  arXiv  Detail & Related papers  (2025-05-18T07:39:17Z)
• CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis [69.02751635551724]
  Spectral imaging offers promising applications across diverse domains, including medicine and urban scene understanding.<n> variability in channel dimensionality and captured wavelengths among spectral cameras impede the development of AI-driven methodologies.<n>We introduce CARL, a model for Camera-Agnostic Representation Learning across RGB, multispectral, and hyperspectral imaging modalities.
  arXiv  Detail & Related papers  (2025-04-27T13:06:40Z)
• SpecDM: Hyperspectral Dataset Synthesis with Pixel-level Semantic Annotations [27.391859339238906]
  In this paper, we explore the potential of generative diffusion model in synthesizing hyperspectral images with pixel-level annotations.<n>To the best of our knowledge, it is the first work to generate high-dimensional HSIs with annotations.<n>We select two of the most widely used dense prediction tasks: semantic segmentation and change detection, and generate datasets suitable for these tasks.
  arXiv  Detail & Related papers  (2025-02-24T11:13:37Z)
• DiffFormer: a Differential Spatial-Spectral Transformer for Hyperspectral Image Classification [3.271106943956333]
  Hyperspectral image classification (HSIC) has gained significant attention because of its potential in analyzing high-dimensional data with rich spectral and spatial information.<n>We propose the Differential Spatial-Spectral Transformer (DiffFormer) to address the inherent challenges of HSIC, such as spectral redundancy and spatial discontinuity.<n>Experiments on benchmark hyperspectral datasets demonstrate the superiority of DiffFormer in terms of classification accuracy, computational efficiency, and generalizability.
  arXiv  Detail & Related papers  (2024-12-23T07:21:41Z)
• HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model [88.13261547704444]
  Hyper SIGMA is a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes.<n>In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images.
  arXiv  Detail & Related papers  (2024-06-17T13:22:58Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.