Cross-Modal RGB-D Fusion Transformer for 6D Pose Estimation of Non-Cooperative Spacecraft with Stereo-Derived Depth

• URL: http://arxiv.org/abs/2605.08592v1
• Date: Sat, 09 May 2026 01:16:33 GMT
• Title: Cross-Modal RGB-D Fusion Transformer for 6D Pose Estimation of Non-Cooperative Spacecraft with Stereo-Derived Depth
• Abstract summary: Learning-based monocular methods have seen widespread adoption in spacecraft pose estimation.<n>They suffer from an intrinsic depth ambiguity problem and tend to fail under the harsh illumination conditions encountered in orbit.<n>Active depth sensors could in principle address the geometric ambiguity, but their power and mass requirements make them poorly suited to most spacecraft platforms.<n>This work addresses these issues through a passive stereo vision framework for six-degree-of-freedom (6-DOF) pose estimation of non-cooperative spacecraft.
• Score: 9.6704752180736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-orbit servicing and active debris removal involving non-cooperative spacecraft require reliable pose estimation to supply accurate position and orientation data for autonomous visual navigation. Learning-based monocular methods have seen widespread adoption in spacecraft pose estimation, yet they suffer from an intrinsic depth ambiguity problem and tend to fail under the harsh illumination conditions routinely encountered in orbit. Active depth sensors could in principle address the geometric ambiguity, but their power and mass requirements make them poorly suited to most spacecraft platforms. This work addresses these issues through a passive stereo vision framework for six-degree-of-freedom (6-DOF) pose estimation of non-cooperative spacecraft. A binocular stereo matching network called TSCA-Stereo is developed to cope with weak-texture surfaces, specular highlights, and severe lighting variations typical of space imagery. A cross-modal fusion Transformer is introduced to combine RGB appearance information with stereo depth features in an adaptive manner, supporting reliable pose recovery. A synthetic binocular multimodal dataset is also built for the experiments, covering stereo disparity maps and 6-DOF pose annotations across a range of illumination scenarios, attitude configurations, and noise levels. Experimental results show that TSCA-Stereo outperforms the baseline across every evaluated metric on this space-specific dataset. The full pose estimation pipeline achieves a mean translation error of 0.0419 m and a mean orientation error of 0.8632° under varied imaging conditions, confirming that the passive stereo approach is both effective and resilient when operating under the demanding visual conditions of the space environment.

Related papers

• Cross-Session 3D LiDAR and Camera Fusion for Robust Localization of Unmanned Aerial Vehicles in GPS-Denied Environments [0.9749189465851389]
  Cross-Fusion is a novel method for real-time UAV localization that integrates data from a 3D Light Detection and Ranging (LiDAR) and a monocular camera.<n>The system employs LiDAR-based odometry for motion tracking and image-based feature matching via a single red-green-blue (RGB) camera to correct drift and improve accuracy.
  arXiv  Detail & Related papers  (2026-06-27T14:48:46Z)
• Precision-Aware Illumination-Disentangled Vision Transformer for Spacecraft 6D Pose Estimation [9.901626173454389]
  This article proposes a Precision-Aware Illumination-Disentangled Vision Transformer (PAID-ViT) for robust spacecraft pose estimation.<n>The proposed model separates pose-relevant structure tokens from illumination-sensitive appearance tokens, estimates patch reliability before pose aggregation, and uses foreground mask supervision to preserve silhouette cues.<n> Experiments on SPEED+ V2, the SPEED+ validation/lightbox/sunlamp evaluation configuration used in this study, suggest that PAID-ViT reduces translation error and improves robustness in the challenging sunlamp domain.
  arXiv  Detail & Related papers  (2026-06-10T03:30:05Z)
• Set-Based Transformer for Atmospheric Compensation in Standoff LWIR Hyperspectral Imaging [6.066442015301664]
  We present a lightweight set-based deep learning framework that takes multiple radiance measurements and estimates transmittance, atmospheric path radiance, and a shared downwelling spectrum.<n>We analyze the learned representation with a sparse autoencoder and observe that several latent features do activate on geographically coherent subsets of the test data.
  arXiv  Detail & Related papers  (2026-06-06T20:25:30Z)
• Unifying UAV Cross-View Geo-Localization via 3D Geometric Perception [51.687842983240564]
  Cross-view geo-localization for Unmanned Aerial Vehicles (UAVs) remains challenging due to the severe geometric discrepancy between oblique UAV imagery and satellite maps.<n>We propose a geometry-aware UAV geo-localization framework that explicitly models the 3D scene geometry to unify coarse place recognition and fine-grained pose estimation.<n>Our method significantly outperforms state-of-the-art baselines, achieving robust meter-level localization accuracy and improved generalization in complex urban environments.
  arXiv  Detail & Related papers  (2026-04-02T08:08:41Z)
• Tightly-Coupled Radar-Visual-Inertial Odometry [12.2300314007703]
  The proposed method fuses image features, radar Doppler measurements, and Inertial Measurement Unit (IMU) measurements within an Iterated Extended Kalman Filter (IEKF) in real-time.<n>The method is evaluated through flight experiments conducted in both indoor and outdoor environments.
  arXiv  Detail & Related papers  (2026-03-24T10:47:04Z)
• Towards Versatile Opti-Acoustic Sensor Fusion and Volumetric Mapping [2.064612766965483]
  Volumetric mapping is critical for autonomous underwater vehicles.<n>vision-based perception provides high-resolution data but fails in turbid conditions.<n> sonar is robust to lighting and turbidity but suffers from low resolution and elevation ambiguity.<n>This paper presents a framework that fuses a stereo sonar pair with a monocular camera to enable safe navigation under varying visibility conditions.
  arXiv  Detail & Related papers  (2026-03-15T16:06:28Z)
• Scale-Aware UAV-to-Satellite Cross-View Geo-Localization: A Semantic Geometric Approach [15.415356946083861]
  Cross-View Geo-Localization between UAV imagery and satellite images plays a crucial role in target localization and UAV self-positioning.<n>Most existing methods rely on the idealized assumption of scale consistency between UAV queries and satellite galleries.<n>This discrepancy leads to field-of-view misalignment and feature mismatch, significantly degrading CVGL robustness.<n>We propose a geometric framework that recovers the absolute metric scale from monocular UAV images using semantic anchors.
  arXiv  Detail & Related papers  (2026-03-08T08:51:19Z)
• VLM6D: VLM based 6Dof Pose Estimation based on RGB-D Images [7.044221981512693]
  VLM6D is a novel dual-stream architecture that leverages the strengths of visual and geometric data from RGB-D input for robust and precise pose estimation.<n>We demonstrate through comprehensive experiments that VLM6D obtained new SOTA performance on the challenging Occluded-LineMOD.
  arXiv  Detail & Related papers  (2025-10-31T05:26:41Z)
• WS-DETR: Robust Water Surface Object Detection through Vision-Radar Fusion with Detection Transformer [4.768265044725289]
  Water surface object detection faces challenges from blurred edges and diverse object scales.<n>Existing approaches suffer from cross-modal feature conflicts, which negatively affect model robustness.<n>We propose a robust vision-radar fusion model WS-DETR, which achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2025-04-10T04:16:46Z)
• Boosting Omnidirectional Stereo Matching with a Pre-trained Depth Foundation Model [70.67610495024459]
  Camera-based setups offer a cost-effective option by using stereo depth estimation to generate dense, high-resolution depth maps.<n>Existing omnidirectional stereo matching approaches achieve only limited depth accuracy across diverse environments.<n>We present DFI-OmniStereo, a novel omnidirectional stereo matching method that leverages a large-scale pre-trained foundation model for relative monocular depth estimation.
  arXiv  Detail & Related papers  (2025-03-30T16:24:22Z)
• Physics-Informed Super-Resolution Diffusion for 6D Phase Space Diagnostics [55.2480439325792]
  An adaptive variational autoencoder embeds initial beam condition images and scalar measurements to a low-dimensional latent space.<n>Projecting from a 6D tensor generates physically consistent 2D projections.<n>Un-supervised adaptive latent space tuning enables tracking of time-varying beams.
  arXiv  Detail & Related papers  (2025-01-08T06:34:32Z)
• Adaptive Stereo Depth Estimation with Multi-Spectral Images Across All Lighting Conditions [58.88917836512819]
  We propose a novel framework incorporating stereo depth estimation to enforce accurate geometric constraints. To mitigate the effects of poor lighting on stereo matching, we introduce Degradation Masking. Our method achieves state-of-the-art (SOTA) performance on the Multi-Spectral Stereo (MS2) dataset.
  arXiv  Detail & Related papers  (2024-11-06T03:30:46Z)
• Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios [103.72094710263656]
  This paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. We propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-02-19T04:39:16Z)
• On Robust Cross-View Consistency in Self-Supervised Monocular Depth Estimation [56.97699793236174]
  We study two kinds of robust cross-view consistency in this paper. We exploit the temporal coherence in both depth feature space and 3D voxel space for self-supervised monocular depth estimation. Experimental results on several outdoor benchmarks show that our method outperforms current state-of-the-art techniques.
  arXiv  Detail & Related papers  (2022-09-19T03:46:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.