Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision

• URL: http://arxiv.org/abs/2411.18025v2
• Date: Mon, 02 Dec 2024 12:42:28 GMT
• Title: Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision
• Abstract summary: We introduce a robotic vision system equipped with pixel-aligned RGB-NIR stereo cameras and a LiDAR sensor mounted on a mobile robot.<n>The system simultaneously captures pixel-aligned pairs of RGB stereo images, NIR stereo images, and temporally synchronized LiDAR points.<n> Experimental results demonstrate the effectiveness of using pixel-aligned RGB-NIR images across diverse lighting conditions.
• Score: 11.112091693344354
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrating RGB and NIR stereo imaging provides complementary spectral information, potentially enhancing robotic 3D vision in challenging lighting conditions. However, existing datasets and imaging systems lack pixel-level alignment between RGB and NIR images, posing challenges for downstream vision tasks. In this paper, we introduce a robotic vision system equipped with pixel-aligned RGB-NIR stereo cameras and a LiDAR sensor mounted on a mobile robot. The system simultaneously captures pixel-aligned pairs of RGB stereo images, NIR stereo images, and temporally synchronized LiDAR points. Utilizing the mobility of the robot, we present a dataset containing continuous video frames under diverse lighting conditions. We then introduce two methods that utilize the pixel-aligned RGB-NIR images: an RGB-NIR image fusion method and a feature fusion method. The first approach enables existing RGB-pretrained vision models to directly utilize RGB-NIR information without fine-tuning. The second approach fine-tunes existing vision models to more effectively utilize RGB-NIR information. Experimental results demonstrate the effectiveness of using pixel-aligned RGB-NIR images across diverse lighting conditions.

Related papers

• Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark [68.49637363691325]
  This paper offers a new perspective for RGB-NIR low-light imaging by incorporating 3D-aware neural modeling.<n>A powerful model can be optimized to implicitly fuse extremely noisy RGB observations with NIR cues in 3D space, effectively recovering clean RGB images.
  arXiv  Detail & Related papers  (2026-07-31T17:59:18Z)
• UniRED: Unified RGB-D Video Frame Interpolation with Event Guidance [19.80142530142947]
  High frame-rate RGB-D videos are crucial for a variety of downstream tasks, including motion analysis, dynamic scene understanding, and 3D reconstruction.<n>Existing video methods have achieved strong performance on RGB data, but they are not readily applicable to RGB-D scenarios.<n>We propose a unified multimodal framework for RGB-D video that jointly exploits RGB appearance, depth geometry, and event-based temporal cues.
  arXiv  Detail & Related papers  (2026-06-23T08:03:58Z)
• Ambient-robust Inverse Rendering using Active RGB-NIR Imaging [22.661549690548544]
  Inverse rendering aims to reconstruct geometry and reflectance of objects from images.<n>We introduce an ambient-robust inverse rendering method enabled by active RGB-NIR imaging.
  arXiv  Detail & Related papers  (2026-05-28T17:13:41Z)
• RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots [11.977612002830725]
  Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction.<n>This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only.
  arXiv  Detail & Related papers  (2026-05-18T10:37:39Z)
• A Multi-View 3D Telepresence System for XR Robot Teleoperation [50.624253593914084]
  effective teleoperation requires intuitive 3D visualization with reliable depth cues.<n>We introduce a multi-view VR telepresence system that fuses geometry from three cameras to produce GPU-accelerated point-cloud rendering on standalone VR hardware.<n>Our pipeline supports real-time rendering of approximately 75k points on the Meta Quest 3.
  arXiv  Detail & Related papers  (2026-04-04T13:22:12Z)
• DepthVision: Robust Vision-Language Understanding through GAN-Based LiDAR-to-RGB Synthesis [11.976362049118782]
  This letter introduces DepthVision, a framework for multimodal scene understanding.<n>It synthesizes RGB images from sparse LiDAR point clouds using a conditional generative adversarial network (GAN)<n>These synthetic views are then combined with real RGB data using a Luminance-Aware Modality Adaptation (LAMA)
  arXiv  Detail & Related papers  (2025-09-09T07:42:07Z)
• DiFuse-Net: RGB and Dual-Pixel Depth Estimation using Window Bi-directional Parallax Attention and Cross-modal Transfer Learning [1.456352735394398]
  DiFuse-Net is a novel decoupled network design for disentangled RGB and DP based depth estimation.<n>WBiPAM captures subtle DP disparity cues unique to smartphone cameras with small aperture.<n>A separate encoder extracts contextual information from the RGB image, and these features are fused to enhance depth prediction.
  arXiv  Detail & Related papers  (2025-06-17T16:49:27Z)
• RASMD: RGB And SWIR Multispectral Driving Dataset for Robust Perception in Adverse Conditions [0.3141085922386211]
  Short-wave infrared (SWIR) imaging offers several advantages over NIR and LWIR. Current autonomous driving algorithms heavily rely on the visible spectrum, which is prone to performance degradation in adverse conditions. We introduce the RGB and SWIR Multispectral Driving dataset, which comprises 100,000 synchronized and spatially aligned RGB-SWIR image pairs.
  arXiv  Detail & Related papers  (2025-04-10T09:54:57Z)
• Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection [67.02804741856512]
  We propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection. Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions.
  arXiv  Detail & Related papers  (2025-01-25T06:21:06Z)
• Pix2Next: Leveraging Vision Foundation Models for RGB to NIR Image Translation [0.536022165180739]
  We propose a novel image-to-image translation framework, Pix2Next, to generate high-quality Near-Infrared (NIR) images from RGB inputs. A multi-scale PatchGAN discriminator ensures realistic image generation at various detail levels, while carefully designed loss functions couple global context understanding with local feature preservation. The proposed approach enables the scaling up of NIR datasets without additional data acquisition or annotation efforts, potentially accelerating advancements in NIR-based computer vision applications.
  arXiv  Detail & Related papers  (2024-09-25T07:51:47Z)
• SEL-CIE: Knowledge-Guided Self-Supervised Learning Framework for CIE-XYZ Reconstruction from Non-Linear sRGB Images [7.932206255996779]
  The CIE-XYZ color space is a device-independent linear space used as part of the camera pipeline. Images are usually saved in non-linear states, and achieving CIE-XYZ color images using conventional methods is not always possible. This paper proposes a framework for using SSL methods alongside paired data to reconstruct CIE-XYZ images and re-render sRGB images, outperforming existing approaches.
  arXiv  Detail & Related papers  (2024-05-20T17:20:41Z)
• UniRGB-IR: A Unified Framework for RGB-Infrared Semantic Tasks via Adapter Tuning [17.36726475620881]
  We propose a general and efficient framework called UniRGB-IR to unify RGB-IR semantic tasks. A novel adapter is developed to efficiently introduce richer RGB-IR features into the pre-trained foundation model. Experimental results on various RGB-IR downstream tasks demonstrate that our method can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-26T12:21:57Z)
• NIR-Assisted Image Denoising: A Selective Fusion Approach and A Real-World Benchmark Dataset [53.79524776100983]
  Leveraging near-infrared (NIR) images to assist visible RGB image denoising shows the potential to address this issue. Existing works still struggle with taking advantage of NIR information effectively for real-world image denoising. We propose an efficient Selective Fusion Module (SFM), which can be plug-and-played into the advanced denoising networks.
  arXiv  Detail & Related papers  (2024-04-12T14:54:26Z)
• DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation [76.81628995237058]
  DFormer is a novel framework to learn transferable representations for RGB-D segmentation tasks. It pretrains the backbone using image-depth pairs from ImageNet-1K. DFormer achieves new state-of-the-art performance on two popular RGB-D tasks.
  arXiv  Detail & Related papers  (2023-09-18T11:09:11Z)
• DarkVisionNet: Low-Light Imaging via RGB-NIR Fusion with Deep Inconsistency Prior [6.162654963520402]
  High-intensity noise in low-light images amplifies the effect of structure inconsistency between RGB-NIR images, which fails existing algorithms. We propose a new RGB-NIR fusion algorithm called Dark Vision Net (DVN) with two technical novelties: Deep Structure and Deep Inconsistency Prior (DIP) Based on the deep structures from both RGB and NIR domains, we introduce the DIP to leverage the structure inconsistency to guide the fusion of RGB-NIR.
  arXiv  Detail & Related papers  (2023-03-13T03:31:29Z)
• Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences [12.799443250845224]
  We propose a novel time-pose function, which is an implicit network that maps timestamps to $rm SE(3)$ elements. Our algorithm consists of three steps: (1) time-pose function fitting, (2) radiance field bootstrapping, (3) joint pose error compensation and radiance field refinement. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drone.
  arXiv  Detail & Related papers  (2022-11-14T15:37:27Z)
• Mirror Complementary Transformer Network for RGB-thermal Salient Object Detection [16.64781797503128]
  RGB-thermal object detection (RGB-T SOD) aims to locate the common prominent objects of an aligned visible and thermal infrared image pair. In this paper, we propose a novel mirror complementary Transformer network (MCNet) for RGB-T SOD. Experiments on benchmark and VT723 datasets show that the proposed method outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-07-07T20:26:09Z)
• Semantic-embedded Unsupervised Spectral Reconstruction from Single RGB Images in the Wild [48.44194221801609]
  We propose a new lightweight and end-to-end learning-based framework to tackle this challenge. We progressively spread the differences between input RGB images and re-projected RGB images from recovered HS images via effective camera spectral response function estimation. Our method significantly outperforms state-of-the-art unsupervised methods and even exceeds the latest supervised method under some settings.
  arXiv  Detail & Related papers  (2021-08-15T05:19:44Z)
• Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection [73.31632581915201]
  We propose a novel data-level recombination strategy to fuse RGB with D (depth) before deep feature extraction. A newly lightweight designed triple-stream network is applied over these novel formulated data to achieve an optimal channel-wise complementary fusion status between the RGB and D.
  arXiv  Detail & Related papers  (2020-08-07T10:13:05Z)
• Synergistic saliency and depth prediction for RGB-D saliency detection [76.27406945671379]
  Existing RGB-D saliency datasets are small, which may lead to overfitting and limited generalization for diverse scenarios. We propose a semi-supervised system for RGB-D saliency detection that can be trained on smaller RGB-D saliency datasets without saliency ground truth.
  arXiv  Detail & Related papers  (2020-07-03T14:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.