Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views

• URL: http://arxiv.org/abs/2507.01835v1
• Date: Wed, 02 Jul 2025 15:49:12 GMT
• Title: Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views
• Authors: Daniil Reutsky, Daniil Vladimirov, Yasin Mamedov, Georgy Perevozchikov, Nancy Mehta, Egor Ershov, Radu Timofte,
• Abstract summary: Hyperspectral reconstruction from RGB images is a fundamentally ill-posed problem due to severe spectral information loss.<n>Existing approaches typically rely on a single RGB image, limiting reconstruction accuracy.<n>We propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system.
• Score: 38.84937150341154
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperspectral reconstruction (HSR) from RGB images is a fundamentally ill-posed problem due to severe spectral information loss. Existing approaches typically rely on a single RGB image, limiting reconstruction accuracy. In this work, we propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system, where two lenses are equipped with carefully selected spectral filters. Our configuration, grounded in theoretical and empirical analysis, enables richer and more diverse spectral observations than conventional single-camera setups. To support this new paradigm, we introduce Doomer, the first dataset for MI-HSR, comprising aligned images from three smartphone cameras and a hyperspectral reference camera across diverse scenes. We show that the proposed HSR model achieves consistent improvements over existing methods on the newly proposed benchmark. In a nutshell, our setup allows 30% towards more accurately estimated spectra compared to an ordinary RGB camera. Our findings suggest that multi-view spectral filtering with commodity hardware can unlock more accurate and practical hyperspectral imaging solutions.

Related papers

• Multispectral Demosaicing via Dual Cameras [29.349707272903835]
  Multispectral (MS) images capture detailed scene information across a wide range of spectral bands.<n>A critical step in processing MS data is demosaicing, which reconstructs color information from the mosaic MS images captured by the camera.<n>This paper proposes a method for MS image demosaicing specifically designed for dual-camera setups where both RGB and MS cameras capture the same scene.
  arXiv  Detail & Related papers  (2025-03-27T22:40:55Z)
• Contourlet Refinement Gate Framework for Thermal Spectrum Distribution Regularized Infrared Image Super-Resolution [54.293362972473595]
  Image super-resolution (SR) aims to reconstruct high-resolution (HR) images from their low-resolution (LR) counterparts. Current approaches to address SR tasks are either dedicated to extracting RGB image features or assuming similar degradation patterns. We propose a Contourlet refinement gate framework to restore infrared modal-specific features while preserving spectral distribution fidelity.
  arXiv  Detail & Related papers  (2024-11-19T14:24:03Z)
• Joint RGB-Spectral Decomposition Model Guided Image Enhancement in Mobile Photography [36.4031416416813]
  We propose a joint RGB-Spectral decomposition model guided enhancement.<n>We leverage thearity between RGB and Low-resolution Multi-Spectral Images (Lr-MSI) to predict shading, reflectance, and material semantic priors.<n>These priors are seamlessly integrated into the established HDRNet to promote dynamic range enhancement, color mapping, and grid expert learning.
  arXiv  Detail & Related papers  (2024-07-25T12:43:41Z)
• psPRF:Pansharpening Planar Neural Radiance Field for Generalized 3D Reconstruction Satellite Imagery [0.6445605125467574]
  Most current NeRF variants for satellites are designed for one specific scene and fall short of generalization to new geometry. This paper introduces psPRF, a Planar Neural Radiance Field designed for paired low-resolution RGB (LR-RGB) and high-resolution panchromatic (HR-PAN) images from satellite sensors with Rational Polynomial Cameras (RPC) To support the generalization ability of psRPF across scenes, we adopt projection loss to ensure strong geometry self-supervision.
  arXiv  Detail & Related papers  (2024-06-22T02:02:32Z)
• Self-Supervised Learning for Real-World Super-Resolution from Dual and Multiple Zoomed Observations [61.448005005426666]
  We consider two challenging issues in reference-based super-resolution (RefSR) for smartphone. We propose a novel self-supervised learning approach for real-world RefSR from observations at dual and multiple camera zooms.
  arXiv  Detail & Related papers  (2024-05-03T15:20:30Z)
• Learning to Recover Spectral Reflectance from RGB Images [20.260831758913902]
  spectral reflectance recovery (SRR) from RGB images is challenging and costly. Most existing approaches are trained on synthetic images and utilize the same parameters for all unseen testing images. We propose a self-supervised meta-auxiliary learning (MAXL) strategy that fine-tunes the well-trained network parameters with each testing image to combine external with internal information.
  arXiv  Detail & Related papers  (2023-04-04T23:27:02Z)
• Semantic-embedded Unsupervised Spectral Reconstruction from Single RGB Images in the Wild [48.44194221801609]
  We propose a new lightweight and end-to-end learning-based framework to tackle this challenge. We progressively spread the differences between input RGB images and re-projected RGB images from recovered HS images via effective camera spectral response function estimation. Our method significantly outperforms state-of-the-art unsupervised methods and even exceeds the latest supervised method under some settings.
  arXiv  Detail & Related papers  (2021-08-15T05:19:44Z)
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB [84.1657998542458]
  It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. This paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery.
  arXiv  Detail & Related papers  (2021-03-26T19:42:21Z)
• Deep Burst Super-Resolution [165.90445859851448]
  We propose a novel architecture for the burst super-resolution task. Our network takes multiple noisy RAW images as input, and generates a denoised, super-resolved RGB image as output. In order to enable training and evaluation on real-world data, we additionally introduce the BurstSR dataset.
  arXiv  Detail & Related papers  (2021-01-26T18:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.