PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation

• URL: http://arxiv.org/abs/2411.15922v3
• Date: Tue, 11 Mar 2025 06:47:38 GMT
• Title: PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation
• Authors: Chia-Ming Lee, Ching-Heng Cheng, Yu-Fan Lin, Yi-Ching Cheng, Wo-Ting Liao, Fu-En Yang, Yu-Chiang Frank Wang, Chih-Chung Hsu,
• Abstract summary: We propose PromptHSI, the first universal AiO HSI restoration framework.<n>Our approach decomposes text prompts into intensity and bias controllers that effectively guide the restoration process.<n>Our architecture excels at both fine-grained recovery and global information restoration across diverse degradation scenarios.
• Score: 28.105125164852367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in All-in-One (AiO) RGB image restoration have demonstrated the effectiveness of prompt learning in handling multiple degradations within a single model. However, extending these approaches to hyperspectral image (HSI) restoration is challenging due to the domain gap between RGB and HSI features, information loss in visual prompts under severe composite degradations, and difficulties in capturing HSI-specific degradation patterns via text prompts. In this paper, we propose PromptHSI, the first universal AiO HSI restoration framework that addresses these challenges. By incorporating frequency-aware feature modulation, which utilizes frequency analysis to narrow down the restoration search space and employing vision-language model (VLM)-guided prompt learning, our approach decomposes text prompts into intensity and bias controllers that effectively guide the restoration process while mitigating domain discrepancies. Extensive experiments demonstrate that our unified architecture excels at both fine-grained recovery and global information restoration across diverse degradation scenarios, highlighting its significant potential for practical remote sensing applications. The source code is available at https://github.com/chingheng0808/PromptHSI.

Related papers

• Beyond Degradation Redundancy: Contrastive Prompt Learning for All-in-One Image Restoration [109.38288333994407]
  Contrastive Prompt Learning (CPL) is a novel framework that fundamentally enhances prompt-task alignment. Our framework establishes new state-of-the-art performance while maintaining parameter efficiency, offering a principled solution for unified image restoration.
  arXiv  Detail & Related papers  (2025-04-14T08:24:57Z)
• VL-UR: Vision-Language-guided Universal Restoration of Images Degraded by Adverse Weather Conditions [3.0133850348026936]
  We propose a vision-language-guided universal restoration framework (VL-UR) VL-UR integrates visual and semantic information to enhance image restoration by integrating visual and semantic information. experiments across eleven diverse degradation settings demonstrate VL-UR's state-of-the-art performance, robustness, and adaptability.
  arXiv  Detail & Related papers  (2025-04-11T02:59:06Z)
• MP-HSIR: A Multi-Prompt Framework for Universal Hyperspectral Image Restoration [15.501904258858112]
  Hyperspectral images (HSIs) often suffer from diverse and unknown degradations during imaging. Existing HSI restoration methods rely on specific degradation assumptions, limiting their effectiveness in complex scenarios. We propose MP-HSIR, a novel multi-prompt framework that effectively integrates spectral, textual, and visual prompts to achieve universal HSI restoration.
  arXiv  Detail & Related papers  (2025-03-12T07:40:49Z)
• UniUIR: Considering Underwater Image Restoration as An All-in-One Learner [49.35128836844725]
  We propose a Universal Underwater Image Restoration method, termed as UniUIR. To decouple degradation-specific issues and explore the inter-correlations among various degradations in UIR task, we designed the Mamba Mixture-of-Experts module. This module extracts degradation prior information in both spatial and frequency domains, and adaptively selects the most appropriate task-specific prompts.
  arXiv  Detail & Related papers  (2025-01-22T16:10:42Z)
• Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration [33.163161549726446]
  Perceive-IR is an all-in-one image restorer designed to achieve fine-grained quality control. In the prompt learning stage, we leverage prompt learning to acquire a fine-grained quality perceiver capable of distinguishing three-tier quality levels. For the restoration stage, a semantic guidance module and compact feature extraction are proposed to further promote the restoration process.
  arXiv  Detail & Related papers  (2024-08-28T17:58:54Z)
• Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration [19.87693298262894]
  We propose Diff-Restorer, a universal image restoration method based on the diffusion model. We utilize the pre-trained visual language model to extract visual prompts from degraded images. We also design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain.
  arXiv  Detail & Related papers  (2024-07-04T05:01:10Z)
• DaLPSR: Leverage Degradation-Aligned Language Prompt for Real-World Image Super-Resolution [19.33582308829547]
  This paper proposes to leverage degradation-aligned language prompt for accurate, fine-grained, and high-fidelity image restoration. The proposed method achieves a new state-of-the-art perceptual quality level.
  arXiv  Detail & Related papers  (2024-06-24T09:30:36Z)
• HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models [38.74983301496911]
  Hyperspectral image (HSI) restoration aims at recovering clean images from degraded observations. Existing model-based methods have limitations in accurately modeling the complex image characteristics. This paper proposes an unsupervised HSI restoration framework with pre-trained diffusion model (HIR-Diff)
  arXiv  Detail & Related papers  (2024-02-24T17:15:05Z)
• SPIRE: Semantic Prompt-Driven Image Restoration [66.26165625929747]
  We develop SPIRE, a Semantic and restoration Prompt-driven Image Restoration framework. Our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength. Our experiments demonstrate the superior restoration performance of SPIRE compared to the state of the arts.
  arXiv  Detail & Related papers  (2023-12-18T17:02:30Z)
• Cross-Consistent Deep Unfolding Network for Adaptive All-In-One Video Restoration [78.14941737723501]
  We propose a Cross-consistent Deep Unfolding Network (CDUN) for All-In-One VR. By orchestrating two cascading procedures, CDUN achieves adaptive processing for diverse degradations. In addition, we introduce a window-based inter-frame fusion strategy to utilize information from more adjacent frames.
  arXiv  Detail & Related papers  (2023-09-04T14:18:00Z)
• DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration [66.01846902242355]
  Blind face restoration usually synthesizes degraded low-quality data with a pre-defined degradation model for training. It is expensive and infeasible to include every type of degradation to cover real-world cases in the training data. We propose Robust Degradation Remover (DR2) to first transform the degraded image to a coarse but degradation-invariant prediction, then employ an enhancement module to restore the coarse prediction to a high-quality image.
  arXiv  Detail & Related papers  (2023-03-13T06:05:18Z)
• HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging [138.04956118993934]
  We propose a high-resolution dual-domain learning network (HDNet) for HSI reconstruction. On the one hand, the proposed HR spatial-spectral attention module with its efficient feature fusion provides continuous and fine pixel-level features. On the other hand, frequency domain learning (FDL) is introduced for HSI reconstruction to narrow the frequency domain discrepancy.
  arXiv  Detail & Related papers  (2022-03-04T06:37:45Z)
• Implicit Subspace Prior Learning for Dual-Blind Face Restoration [66.67059961379923]
  A novel implicit subspace prior learning (ISPL) framework is proposed as a generic solution to dual-blind face restoration. Experimental results demonstrate significant perception-distortion improvement of ISPL against existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-12T08:04:24Z)
• HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment [63.333407973913374]
  "Face Renovation"(FR) is a semantic-guided generation problem. "HiFaceGAN" is a multi-stage framework containing several nested CSR units. experiments on both synthetic and real face images have verified the superior performance of HiFaceGAN.
  arXiv  Detail & Related papers  (2020-05-11T11:33:17Z)
• Real-world Person Re-Identification via Degradation Invariance Learning [111.86722193694462]
  Person re-identification (Re-ID) in real-world scenarios usually suffers from various degradation factors, e.g., low-resolution, weak illumination, blurring and adverse weather. We propose a degradation invariance learning framework for real-world person Re-ID. By introducing a self-supervised disentangled representation learning strategy, our method is able to simultaneously extract identity-related robust features.
  arXiv  Detail & Related papers  (2020-04-10T07:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.