Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimation

• URL: http://arxiv.org/abs/2409.06240v1
• Date: Tue, 10 Sep 2024 06:17:07 GMT
• Title: Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimation
• Authors: Mohsi Jawaid, Rajat Talak, Yasir Latif, Luca Carlone, Tat-Jun Chin,
• Abstract summary: A major cause of the Sim2Real gap are novel lighting conditions encountered during test time. The paper proposes a test-time self-supervision scheme with a certifier module.
• Score: 39.53065184075108
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning plays a critical role in vision-based satellite pose estimation. However, the scarcity of real data from the space environment means that deep models need to be trained using synthetic data, which raises the Sim2Real domain gap problem. A major cause of the Sim2Real gap are novel lighting conditions encountered during test time. Event sensors have been shown to provide some robustness against lighting variations in vision-based pose estimation. However, challenging lighting conditions due to strong directional light can still cause undesirable effects in the output of commercial off-the-shelf event sensors, such as noisy/spurious events and inhomogeneous event densities on the object. Such effects are non-trivial to simulate in software, thus leading to Sim2Real gap in the event domain. To close the Sim2Real gap in event-based satellite pose estimation, the paper proposes a test-time self-supervision scheme with a certifier module. Self-supervision is enabled by an optimisation routine that aligns a dense point cloud of the predicted satellite pose with the event data to attempt to rectify the inaccurately estimated pose. The certifier attempts to verify the corrected pose, and only certified test-time inputs are backpropagated via implicit differentiation to refine the predicted landmarks, thus improving the pose estimates and closing the Sim2Real gap. Results show that the our method outperforms established test-time adaptation schemes.

Related papers

• Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing [2.194575078433007]
  Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic
  arXiv  Detail & Related papers  (2024-04-29T10:37:38Z)
• Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap [6.393953433174051]
  We propose a novel perspective for addressing the real-to-simulated data gap. We conduct the first large-scale investigation into the real-to-simulated data gap in an autonomous driving setting. Our results show notable improvements in model robustness to simulated data, even improving real-world performance in some cases.
  arXiv  Detail & Related papers  (2024-03-24T11:09:41Z)
• LROC-PANGU-GAN: Closing the Simulation Gap in Learning Crater Segmentation with Planetary Simulators [5.667566032625522]
  It is critical for probes landing on foreign planetary bodies to be able to robustly identify and avoid hazards. Recent applications of deep learning to this problem show promising results. These models are, however, often learned with explicit supervision over annotated datasets. This paper introduces a system to close this "realism" gap while retaining label fidelity.
  arXiv  Detail & Related papers  (2023-10-04T12:52:38Z)
• Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting [72.75792823726479]
  We propose Conditional Domain Translation via Conditional Alignment and Reweighting (CARE) to close the sim2real appearance and content gaps. We present an analytical justification of our algorithm and demonstrate strong gains over competing methods on standard benchmarks.
  arXiv  Detail & Related papers  (2023-02-09T18:39:28Z)
• Towards Spatial Equilibrium Object Detection [88.9747319572368]
  In this paper, we study the spatial disequilibrium problem of modern object detectors. We propose to quantify this problem by measuring the detection performance over zones. This motivates us to design a more generalized measurement, termed Spatial equilibrium Precision.
  arXiv  Detail & Related papers  (2023-01-14T17:33:26Z)
• Practical Exposure Correction: Great Truths Are Always Simple [65.82019845544869]
  We establish a Practical Exposure Corrector (PEC) that assembles the characteristics of efficiency and performance. We introduce an exposure adversarial function as the key engine to fully extract valuable information from the observation. Our experiments fully reveal the superiority of our proposed PEC.
  arXiv  Detail & Related papers  (2022-12-29T09:52:13Z)
• Towards Bridging the Space Domain Gap for Satellite Pose Estimation using Event Sensing [35.467052373502575]
  Event sensing offers a promising solution to generalise from the simulation to the target domain under stark illumination differences. Our main contribution is an event-based satellite pose estimation technique, trained purely on synthetic data. Results on the dataset showed that our event-based satellite pose estimation method, trained only on synthetic data without adaptation, could generalise to the target domain effectively.
  arXiv  Detail & Related papers  (2022-09-24T07:22:09Z)
• Ev-TTA: Test-Time Adaptation for Event-Based Object Recognition [7.814941658661939]
  Ev-TTA is a simple, effective test-time adaptation for event-based object recognition. Our formulation can be successfully applied regardless of input representations and extended into regression tasks.
  arXiv  Detail & Related papers  (2022-03-23T07:43:44Z)
• Motion Deblurring with Real Events [50.441934496692376]
  We propose an end-to-end learning framework for event-based motion deblurring in a self-supervised manner. Real-world events are exploited to alleviate the performance degradation caused by data inconsistency.
  arXiv  Detail & Related papers  (2021-09-28T13:11:44Z)
• Worsening Perception: Real-time Degradation of Autonomous Vehicle Perception Performance for Simulation of Adverse Weather Conditions [47.529411576737644]
  This study explores the potential of using a simple, lightweight image augmentation system in an autonomous racing vehicle. With minimal adjustment, the prototype system can replicate the effects of both water droplets on the camera lens, and fading light conditions.
  arXiv  Detail & Related papers  (2021-03-03T23:49:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.