Multispectral Demosaicing via Dual Cameras

• URL: http://arxiv.org/abs/2503.22026v2
• Date: Wed, 09 Apr 2025 00:18:02 GMT
• Title: Multispectral Demosaicing via Dual Cameras
• Authors: SaiKiran Tedla, Junyong Lee, Beixuan Yang, Mahmoud Afifi, Michael S. Brown,
• Abstract summary: Multispectral (MS) images capture detailed scene information across a wide range of spectral bands.<n>A critical step in processing MS data is demosaicing, which reconstructs color information from the mosaic MS images captured by the camera.<n>This paper proposes a method for MS image demosaicing specifically designed for dual-camera setups where both RGB and MS cameras capture the same scene.
• Score: 29.349707272903835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multispectral (MS) images capture detailed scene information across a wide range of spectral bands, making them invaluable for applications requiring rich spectral data. Integrating MS imaging into multi camera devices, such as smartphones, has the potential to enhance both spectral applications and RGB image quality. A critical step in processing MS data is demosaicing, which reconstructs color information from the mosaic MS images captured by the camera. This paper proposes a method for MS image demosaicing specifically designed for dual-camera setups where both RGB and MS cameras capture the same scene. Our approach leverages co-captured RGB images, which typically have higher spatial fidelity, to guide the demosaicing of lower-fidelity MS images. We introduce the Dual-camera RGB-MS Dataset - a large collection of paired RGB and MS mosaiced images with ground-truth demosaiced outputs - that enables training and evaluation of our method. Experimental results demonstrate that our method achieves state-of-the-art accuracy compared to existing techniques.

Related papers

• Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views [38.84937150341154]
  Hyperspectral reconstruction from RGB images is a fundamentally ill-posed problem due to severe spectral information loss.<n>Existing approaches typically rely on a single RGB image, limiting reconstruction accuracy.<n>We propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system.
  arXiv  Detail & Related papers  (2025-07-02T15:49:12Z)
• DiFuse-Net: RGB and Dual-Pixel Depth Estimation using Window Bi-directional Parallax Attention and Cross-modal Transfer Learning [1.456352735394398]
  DiFuse-Net is a novel decoupled network design for disentangled RGB and DP based depth estimation.<n>WBiPAM captures subtle DP disparity cues unique to smartphone cameras with small aperture.<n>A separate encoder extracts contextual information from the RGB image, and these features are fused to enhance depth prediction.
  arXiv  Detail & Related papers  (2025-06-17T16:49:27Z)
• Image Demoiréing Using Dual Camera Fusion on Mobile Phones [58.39212652291496]
  We propose to utilize Dual Camera fusion for Image Demoir'eing (DCID), ie, using the ultra-wide-angle (UW) image to assist the moir'e removal of wide-angle (W) image.<n>In particular, we propose an efficient DCID method, where a lightweight UW image encoder is integrated into an existing demoir'eing network.
  arXiv  Detail & Related papers  (2025-06-10T02:20:37Z)
• Limitations of Data-Driven Spectral Reconstruction -- Optics-Aware Analysis and Mitigation [22.07699685165064]
  Recent efforts in data-driven spectral reconstruction aim at extracting spectral information from RGB images captured by cost-effective RGB cameras.<n>We evaluate both the practical limitations with respect to current datasets and overfitting, as well as fundamental limitations with respect to the nature of the information encoded in the RGB images.<n>We propose to exploit the combination of metameric data augmentation and optical lens aberrations to improve the encoding of the metameric information into the RGB image.
  arXiv  Detail & Related papers  (2024-01-08T11:46:45Z)
• EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion [55.367269556557645]
  EvPlug learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model. We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation.
  arXiv  Detail & Related papers  (2023-12-28T10:05:13Z)
• Image Demoireing in RAW and sRGB Domains [18.921026683632146]
  We develop Skip-Connection-based Demoireing Module (SCDM) with Gated Feedback Module (GFM) and Frequency Selection Module (FSM) We design a RGB Guided ISP (RGISP) to learn a device-dependent ISP, assisting the process of color recovery. Our RRID outperforms state-of-the-art approaches, in terms of the performance in moire pattern removal and color cast correction by 0.62dB in PSNR and 0.003 in SSIM.
  arXiv  Detail & Related papers  (2023-12-14T16:00:28Z)
• mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection [14.052943954940758]
  This work introduces a new multispectral database and novel approaches for eyeblink detection in RGB and Near-Infrared (NIR) individual images. mEBAL2 is the largest existing eyeblink database. mEBAL2 includes 21,100 image sequences from 180 different students.
  arXiv  Detail & Related papers  (2023-09-14T17:25:25Z)
• Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution [52.582632746409665]
  We propose a novel Symmetric Uncertainty-aware Feature Transmission (SUFT) for color-guided DSR. Our method achieves superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-01T06:35:59Z)
• ObjectFormer for Image Manipulation Detection and Localization [118.89882740099137]
  We propose ObjectFormer to detect and localize image manipulations. We extract high-frequency features of the images and combine them with RGB features as multimodal patch embeddings. We conduct extensive experiments on various datasets and the results verify the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-03-28T12:27:34Z)
• Multi-Scale Iterative Refinement Network for RGB-D Salient Object Detection [7.062058947498447]
  salient visual cues appear in various scales and resolutions of RGB images due to semantic gaps at different feature levels. Similar salient patterns are available in cross-modal depth images as well as multi-scale versions. We devise attention based fusion module (ABF) to address on cross-modal correlation.
  arXiv  Detail & Related papers  (2022-01-24T10:33:00Z)
• Dual-Camera Super-Resolution with Aligned Attention Modules [56.54073689003269]
  We present a novel approach to reference-based super-resolution (RefSR) with the focus on dual-camera super-resolution (DCSR) Our proposed method generalizes the standard patch-based feature matching with spatial alignment operations. To bridge the domain gaps between real-world images and the training images, we propose a self-supervised domain adaptation strategy.
  arXiv  Detail & Related papers  (2021-09-03T07:17:31Z)
• Semantic-embedded Unsupervised Spectral Reconstruction from Single RGB Images in the Wild [48.44194221801609]
  We propose a new lightweight and end-to-end learning-based framework to tackle this challenge. We progressively spread the differences between input RGB images and re-projected RGB images from recovered HS images via effective camera spectral response function estimation. Our method significantly outperforms state-of-the-art unsupervised methods and even exceeds the latest supervised method under some settings.
  arXiv  Detail & Related papers  (2021-08-15T05:19:44Z)
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation [62.210091681352914]
  We study multi-sensor fusion for 3D semantic segmentation for many applications, such as autonomous driving and robotics. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) We propose a two-stream network to extract features from the two modalities separately. The extracted features are fused by effective residual-based fusion modules.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• Enhanced Hyperspectral Image Super-Resolution via RGB Fusion and TV-TV Minimization [9.584717030078245]
  Hyperspectral (HS) images contain detailed spectral information that has proven crucial in applications like remote sensing, surveillance, and astronomy. Because of hardware limitations of HS cameras, the captured images have low spatial resolution. To improve them, the low-resolution hyperspectral images are fused with conventional high-resolution RGB images via a technique known as fusion based HS image super-resolution.
  arXiv  Detail & Related papers  (2021-06-13T18:52:47Z)
• Deep Burst Super-Resolution [165.90445859851448]
  We propose a novel architecture for the burst super-resolution task. Our network takes multiple noisy RAW images as input, and generates a denoised, super-resolved RGB image as output. In order to enable training and evaluation on real-world data, we additionally introduce the BurstSR dataset.
  arXiv  Detail & Related papers  (2021-01-26T18:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.