Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements

• URL: http://arxiv.org/abs/2603.22182v1
• Date: Mon, 23 Mar 2026 16:45:03 GMT
• Title: Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements
• Abstract summary: We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency.<n>The learned representations are integrated with a Reinforcement Learning-based policy for collision-free navigation in unstructured environments.
• Score: 9.487836148196713
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a cross-modal learning framework that exploits complementary information from depth and grayscale images for robust navigation. We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency, enabling the system to infer depth-relevant features from grayscale observations when depth measurements are corrupted. The learned representations are integrated with a Reinforcement Learning-based policy for collision-free navigation in unstructured environments when depth sensors experience degradation due to adverse conditions such as poor lighting or reflective surfaces. Simulation and real-world experiments demonstrate that our approach maintains robust performance under significant depth degradation and successfully transfers to real environments.

Related papers

• Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning [45.2217589644315]
  We propose a sensor-guided adaptive contrastive learning framework for visual representation learning in PointGoal navigation.<n>During training, privileged LiDAR sensing guides the contrastive objective through a geometry-aware similarity metric and adaptive temperature scaling.<n>The resulting encoder is pretrained independently, frozen, and used as the backbone for reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-03T23:15:50Z)
• NAIMA: Semantics Aware RGB Guided Depth Super-Resolution [50.15623093332659]
  Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details.<n>We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings.
  arXiv  Detail & Related papers  (2026-04-06T04:14:59Z)
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning [59.88543114325153]
  We introduce the Seeing-to-Experiencing framework to scale the capability of navigation foundation models with reinforcement learning.<n>S2E combines the strengths of pre-training on videos and post-training through RL.<n>We establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3DGS reconstructions of real-world scenes.
  arXiv  Detail & Related papers  (2025-07-29T17:26:10Z)
• DCIRNet: Depth Completion with Iterative Refinement for Dexterous Grasping of Transparent and Reflective Objects [9.235004977824026]
  We propose DCIRNet, a novel multimodal depth completion network for transparent and reflective objects.<n>Our approach incorporates an innovative multimodal feature fusion module designed to extract complementary information between RGB images and incomplete depth maps.<n>We achieve a $44%$ improvement in the grasp success rate for transparent and reflective objects.
  arXiv  Detail & Related papers  (2025-06-11T08:04:22Z)
• Back to the Color: Learning Depth to Specific Color Transformation for Unsupervised Depth Estimation [45.07558105128673]
  discrepancies between synthetic and real-world colors pose significant challenges for depth estimation in real-world scenes. We propose Back2Color, a framework that predicts realistic colors from depth using a model trained on real-world data. We also present VADepth, based on the Vision Attention Network, which offers lower computational complexity and higher accuracy than transformers.
  arXiv  Detail & Related papers  (2024-06-11T21:55:20Z)
• Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios [103.72094710263656]
  This paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. We propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-02-19T04:39:16Z)
• Depth-aware Volume Attention for Texture-less Stereo Matching [67.46404479356896]
  We propose a lightweight volume refinement scheme to tackle the texture deterioration in practical outdoor scenarios. We introduce a depth volume supervised by the ground-truth depth map, capturing the relative hierarchy of image texture. Local fine structure and context are emphasized to mitigate ambiguity and redundancy during volume aggregation.
  arXiv  Detail & Related papers  (2024-02-14T04:07:44Z)
• Structure Flow-Guided Network for Real Depth Super-Resolution [28.63334760296165]
  We propose a novel structure flow-guided depth super-resolution (DSR) framework. A cross-modality flow map is learned to guide the RGB-structure information transferring for precise depth upsampling. Our framework achieves excellent performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-01-31T05:13:55Z)
• Learning Deep Context-Sensitive Decomposition for Low-Light Image Enhancement [58.72667941107544]
  A typical framework is to simultaneously estimate the illumination and reflectance, but they disregard the scene-level contextual information encapsulated in feature spaces. We develop a new context-sensitive decomposition network architecture to exploit the scene-level contextual dependencies on spatial scales. We develop a lightweight CSDNet (named LiteCSDNet) by reducing the number of channels.
  arXiv  Detail & Related papers  (2021-12-09T06:25:30Z)
• Self-Guided Instance-Aware Network for Depth Completion and Enhancement [6.319531161477912]
  Existing methods directly interpolate the missing depth measurements based on pixel-wise image content and the corresponding neighboring depth values. We propose a novel self-guided instance-aware network (SG-IANet) that utilize self-guided mechanism to extract instance-level features that is needed for depth restoration.
  arXiv  Detail & Related papers  (2021-05-25T19:41:38Z)
• Channel Attention based Iterative Residual Learning for Depth Map Super-Resolution [58.626803922196146]
  We argue that DSR models trained on synthetic dataset are restrictive and not effective in dealing with real-world DSR tasks. We make two contributions in tackling real-world degradation of different depth sensors. We propose a new framework for real-world DSR, which consists of four modules.
  arXiv  Detail & Related papers  (2020-06-02T09:12:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.