HyperKD: Distilling Cross-Spectral Knowledge in Masked Autoencoders via Inverse Domain Shift with Spatial-Aware Masking and Specialized Loss

• URL: http://arxiv.org/abs/2508.09453v1
• Date: Wed, 13 Aug 2025 03:10:40 GMT
• Title: HyperKD: Distilling Cross-Spectral Knowledge in Masked Autoencoders via Inverse Domain Shift with Spatial-Aware Masking and Specialized Loss
• Authors: Abdul Matin, Tanjim Bin Faruk, Shrideep Pallickara, Sangmi Lee Pallickara,
• Abstract summary: HyperKD is a novel knowledge distillation framework that enables transfer learned representations from a teacher model into a student model for effective development of a foundation model on hyperspectral images.<n>Building upon a Masked Autoencoder, HyperKD distills knowledge from the Prithvi foundational model into a student tailored for EnMAP hyperspectral imagery.
• Score: 1.1999555634662633
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The proliferation of foundation models, pretrained on large-scale unlabeled datasets, has emerged as an effective approach in creating adaptable and reusable architectures that can be leveraged for various downstream tasks using satellite observations. However, their direct application to hyperspectral remote sensing remains challenging due to inherent spectral disparities and the scarcity of available observations. In this work, we present HyperKD, a novel knowledge distillation framework that enables transferring learned representations from a teacher model into a student model for effective development of a foundation model on hyperspectral images. Unlike typical knowledge distillation frameworks, which use a complex teacher to guide a simpler student, HyperKD enables an inverse form of knowledge transfer across different types of spectral data, guided by a simpler teacher model. Building upon a Masked Autoencoder, HyperKD distills knowledge from the Prithvi foundational model into a student tailored for EnMAP hyperspectral imagery. HyperKD addresses the inverse domain adaptation problem with spectral gaps by introducing a feature-based strategy that includes spectral range-based channel alignment, spatial feature-guided masking, and an enhanced loss function tailored for hyperspectral images. HyperKD bridges the substantial spectral domain gap, enabling the effective use of pretrained foundation models for geospatial applications. Extensive experiments show that HyperKD significantly improves representation learning in MAEs, leading to enhanced reconstruction fidelity and more robust performance on downstream tasks such as land cover classification, crop type identification, and soil organic carbon prediction, underpinning the potential of knowledge distillation frameworks in remote sensing analytics with hyperspectral imagery.

Related papers

• SDHSI-Net: Learning Better Representations for Hyperspectral Images via Self-Distillation [20.532231645997864]
  Hyperspectral image (HSI) classification presents unique challenges due to its high spectral dimensionality and limited labeled data.<n>Traditional deep learning models often suffer from overfitting and high computational costs.<n>Self-distillation (SD) is a variant of knowledge distillation where a network learns from its own predictions.
  arXiv  Detail & Related papers  (2026-01-12T10:57:24Z)
• Label-Efficient Hyperspectral Image Classification via Spectral FiLM Modulation of Low-Level Pretrained Diffusion Features [22.476787402621138]
  We present a label-efficient framework that leverages spatial features from a frozen diffusion model pretrained on natural images.<n>Our results indicate that pretrained diffusion models can support domain-agnostic, label-efficient representation learning.
  arXiv  Detail & Related papers  (2025-12-03T04:23:54Z)
• Adversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks [61.608453110751206]
  We propose a fast and high-quality data-free knowledge distillation approach for graph neural networks.<n>The proposed graph-free KD method (ACGKD) significantly reduces the spatial complexity of pseudo-graphs.<n>ACGKD eliminates the dimensional ambiguity between the student and teacher models by increasing the student's dimensions.
  arXiv  Detail & Related papers  (2025-04-01T08:44:27Z)
• Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data [14.104497777255137]
  We introduce Low-rank Efficient Spatial-Spectral Vision Transformer with three key innovations.<n>We pretrain LESS ViT using a Hyperspectral Masked Autoencoder framework with integrated positional and channel masking strategies.<n> Experimental results demonstrate that our proposed method achieves competitive performance against state-of-the-art multi-modal geospatial foundation models.
  arXiv  Detail & Related papers  (2025-03-17T05:42:19Z)
• Spatial-Spectral Diffusion Contrastive Representation Network for Hyperspectral Image Classification [8.600534616819333]
  This paper presents a Spatial-Spectral Diffusion Contrastive Representation Network (DiffCRN)<n>DiffCRN is based on denoising diffusion probabilistic model (DDPM) combined with contrastive learning (CL) for hyperspectral images classification.<n> Experiments conducted on widely used four HSI datasets demonstrate the improved performance of the proposed DiffCRN.
  arXiv  Detail & Related papers  (2025-02-27T02:34:23Z)
• Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillation [52.0297393822012]
  We introduce an assistant model as a bridge to facilitate smooth feature knowledge transfer between heterogeneous teachers and students.<n>Within our proposed design principle, the assistant model combines the advantages of cross-architecture inductive biases and module functions.<n>Our proposed method is evaluated across some homogeneous model pairs and arbitrary heterogeneous combinations of CNNs, ViTs, spatial KDs.
  arXiv  Detail & Related papers  (2024-10-16T08:02:49Z)
• HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model [88.13261547704444]
  Hyper SIGMA is a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes.<n>In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images.
  arXiv  Detail & Related papers  (2024-06-17T13:22:58Z)
• DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection [55.48770333927732]
  We propose a Difusion-based Anomaly Detection (DiAD) framework for multi-class anomaly detection. It consists of a pixel-space autoencoder, a latent-space Semantic-Guided (SG) network with a connection to the stable diffusion's denoising network, and a feature-space pre-trained feature extractor. Experiments on MVTec-AD and VisA datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-12-11T18:38:28Z)
• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation [69.65734716679925]
  Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family. We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
  arXiv  Detail & Related papers  (2023-10-30T11:13:02Z)
• Anomaly Detection via Reverse Distillation from One-Class Embedding [2.715884199292287]
  We propose a novel T-S model consisting of a teacher encoder and a student decoder. Instead of receiving raw images directly, the student network takes teacher model's one-class embedding as input. In addition, we introduce a trainable one-class bottleneck embedding module in our T-S model.
  arXiv  Detail & Related papers  (2022-01-26T01:48:37Z)
• A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration [36.525810477650026]
  Hyperspectral imaging offers new perspectives for diverse applications. The lack of accurate ground-truth "clean" hyperspectral signals on the spot makes restoration tasks challenging. In this paper, we advocate for a hybrid approach based on sparse coding principles.
  arXiv  Detail & Related papers  (2021-11-18T14:16:04Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.