HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone

• URL: http://arxiv.org/abs/2605.17286v1
• Date: Sun, 17 May 2026 06:50:06 GMT
• Title: HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone
• Abstract summary: We propose HyperVision, the first ground-based hyperspectral pre-trained backbone.<n>It is pre-trained on a collection of 15k images from 26 diverse ground-based datasets.<n>It achieves state-of-the-art performance compared to task-specific methods.
• Score: 20.52608763146186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While hyperspectral imaging provides rich spatial-spectral information across hundreds of narrow wavelength bands for precise material identification, ground-based hyperspectral pre-trained backbones remain absent, constrained by varying spectral configurations across sensors, the scarcity and inconsistency of labels, and the limited scale and scene diversity of existing datasets. To address these challenges and enable universal perception, we propose HyperVision, the first ground-based hyperspectral pre-trained backbone. First, to handle varying spectral configurations, HyperVision adopts a channel-adaptive dynamic embedding mechanism to map heterogeneous inputs into a unified token space. Second, to address the scarcity and inconsistency of labels, we introduce a multi-source pseudo-labeling method that fuses semantic representations from both spatial structures generated by SAM2 and fine-grained spectral material information extracted by HyperFree. Third, to compensate for limited dataset scale and enrich scene diversity, a cross-modal knowledge distillation mechanism is utilized to transfer rich semantic representations from a pre-trained RGB vision model to our hyperspectral backbone. Pre-trained on a collection of 15k images from 26 diverse ground-based datasets, HyperVision demonstrates exceptional generalization. Requiring only efficient head-only adaptation without adjusting backbone parameters, it achieves state-of-the-art performance compared to task-specific methods across three downstream tasks under varying sensor configurations, yielding up to a 16.3% relative improvement in hyperspectral semantic segmentation $\mathrm{Acc}_{\mathrm{M}}$, a 2.1% relative gain in object tracking AUC, and a 35.5% reduction in salient object detection MAE. The source code and pre-trained model will be publicly available at https://github.com/lronkitty/HyperVision .

Related papers

• MSP-Net: Manifold-Guided Spectral Prompt Network for Hyperspectral Object Tracking [11.03462046191901]
  Hyperspectral object tracking offers unique advantages for target discrimination in complex scenes.<n>Existing methods treat hyperspectral images as multi-channel extensions of RGB images, performing feature fusion in fixed band order.<n>We propose the Manifold-Guided Spectral Prompt Network (MSP-Net)<n> Experiments on HOT2020 and HOT2023 demonstrate MSP-Net achieves AUC and Precision exceeding 0.80 and 0.96, respectively.
  arXiv  Detail & Related papers  (2026-08-10T13:09:25Z)
• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence [78.1406635199656]
  Holi-Spatial is the first fully automated, large-scale, spatially-aware multimodal dataset constructed from raw video inputs without human intervention.<n>Holi-Spatial-4M is the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs.
  arXiv  Detail & Related papers  (2026-03-08T14:49:20Z)
• SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping [9.392313789126714]
  SpecAware is a novel hyperspectral spectral-content aware foundation model for unifying multi-sensor learning for HSI mapping.<n>The core of SpecAware is a two-step hypernetwork-driven encoding process for HSI data.<n>Experiments on six datasets demonstrate that SpecAware can learn superior feature representations.
  arXiv  Detail & Related papers  (2025-10-31T06:28:14Z)
• Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models [18.24287471339871]
  Hyperspectral imaging (HSI) captures spatial information along with dense spectral measurements across numerous narrow wavelength bands.<n>Our architecture incorporates a spectral transformer and a spectrum-aware spatial prior module to extract rich spatial-spectral features.<n>Our architecture achieves state-of-the-art semantic segmentation performance while directly using HSI inputs, outperforming both vision-based and hyperspectral segmentation methods.
  arXiv  Detail & Related papers  (2025-09-24T13:32:07Z)
• Hyperspectral Mamba for Hyperspectral Object Tracking [56.365517163296936]
  A new hyperspectral object tracking network equipped with Mamba (HyMamba) is proposed.<n>It unifies spectral, cross-depth, and temporal modeling through state space modules (SSMs)<n>HyMamba achieves state-of-the-art performance on seven benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-10T03:47:43Z)
• AuxDet: Auxiliary Metadata Matters for Omni-Domain Infrared Small Target Detection [49.81255045696323]
  We present the Auxiliary Metadata Driven Infrared Small Target Detector (AuxDet)<n>AuxDet integrates metadata semantics with visual features, guiding adaptive representation learning for each sample.<n>Experiments on the challenging WideIRSTD-Full benchmark demonstrate that AuxDet consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-21T07:02:05Z)
• DiffFormer: a Differential Spatial-Spectral Transformer for Hyperspectral Image Classification [3.271106943956333]
  Hyperspectral image classification (HSIC) has gained significant attention because of its potential in analyzing high-dimensional data with rich spectral and spatial information.<n>We propose the Differential Spatial-Spectral Transformer (DiffFormer) to address the inherent challenges of HSIC, such as spectral redundancy and spatial discontinuity.<n>Experiments on benchmark hyperspectral datasets demonstrate the superiority of DiffFormer in terms of classification accuracy, computational efficiency, and generalizability.
  arXiv  Detail & Related papers  (2024-12-23T07:21:41Z)
• AMBER -- Advanced SegFormer for Multi-Band Image Segmentation: an application to Hyperspectral Imaging [0.0]
  This paper introduces AMBER, an advanced SegFormer specifically designed for multi-band image segmentation.<n>AMBER enhances the original SegFormer by incorporating three-dimensional convolutions, custom kernel sizes, and a Funnelizer layer.<n>Our experiments, conducted on three benchmark datasets and on a dataset from the PRISMA satellite, show that AMBER outperforms traditional CNN-based methods in terms of Overall Accuracy, Kappa coefficient, and Average Accuracy.
  arXiv  Detail & Related papers  (2024-09-14T09:34:05Z)
• HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model [88.13261547704444]
  Hyper SIGMA is a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes.<n>In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images.
  arXiv  Detail & Related papers  (2024-06-17T13:22:58Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• Object Detection in Hyperspectral Image via Unified Spectral-Spatial Feature Aggregation [55.9217962930169]
  We present S2ADet, an object detector that harnesses the rich spectral and spatial complementary information inherent in hyperspectral images. S2ADet surpasses existing state-of-the-art methods, achieving robust and reliable results.
  arXiv  Detail & Related papers  (2023-06-14T09:01:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.