Spatiotemporal Contrastive Learning for Cross-View Video Localization in Unstructured Off-road Terrains

• URL: http://arxiv.org/abs/2506.05250v1
• Date: Thu, 05 Jun 2025 17:10:29 GMT
• Title: Spatiotemporal Contrastive Learning for Cross-View Video Localization in Unstructured Off-road Terrains
• Authors: Zhiyun Deng, Dongmyeong Lee, Amanda Adkins, Jesse Quattrociocchi, Christian Ellis, Joydeep Biswas,
• Abstract summary: MoViX is a self-supervised cross-view video localization framework.<n>It learns viewpoint- and season-invariant representations while preserving directional awareness.<n>MoViX localizes within 25 meters of ground truth 93% of the time and within 50 meters 100% of the time in unseen regions.
• Score: 6.8857090684309155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robust cross-view 3-DoF localization in GPS-denied, off-road environments remains challenging due to (1) perceptual ambiguities from repetitive vegetation and unstructured terrain, and (2) seasonal shifts that significantly alter scene appearance, hindering alignment with outdated satellite imagery. To address this, we introduce MoViX, a self-supervised cross-view video localization framework that learns viewpoint- and season-invariant representations while preserving directional awareness essential for accurate localization. MoViX employs a pose-dependent positive sampling strategy to enhance directional discrimination and temporally aligned hard negative mining to discourage shortcut learning from seasonal cues. A motion-informed frame sampler selects spatially diverse frames, and a lightweight temporal aggregator emphasizes geometrically aligned observations while downweighting ambiguous ones. At inference, MoViX runs within a Monte Carlo Localization framework, using a learned cross-view matching module in place of handcrafted models. Entropy-guided temperature scaling enables robust multi-hypothesis tracking and confident convergence under visual ambiguity. We evaluate MoViX on the TartanDrive 2.0 dataset, training on under 30 minutes of data and testing over 12.29 km. Despite outdated satellite imagery, MoViX localizes within 25 meters of ground truth 93% of the time, and within 50 meters 100% of the time in unseen regions, outperforming state-of-the-art baselines without environment-specific tuning. We further demonstrate generalization on a real-world off-road dataset from a geographically distinct site with a different robot platform.

Related papers

• STRMs: Spatial Temporal Reasoning Models for Vision-Based Localization Rivaling GPS Precision [3.671692919685993]
  We introduce two sequential generative models, VAE-RNN and VAE-Transformer, which transform first-person perspective observations into global map perspective representations.<n>We evaluate these models across two real-world environments: a university campus navigated by a Jackal robot and an urban downtown area navigated by a Tesla sedan.
  arXiv  Detail & Related papers  (2025-03-11T00:38:54Z)
• Neural Semantic Map-Learning for Autonomous Vehicles [85.8425492858912]
  We present a mapping system that fuses local submaps gathered from a fleet of vehicles at a central instance to produce a coherent map of the road environment. Our method jointly aligns and merges the noisy and incomplete local submaps using a scene-specific Neural Signed Distance Field. We leverage memory-efficient sparse feature-grids to scale to large areas and introduce a confidence score to model uncertainty in scene reconstruction.
  arXiv  Detail & Related papers  (2024-10-10T10:10:03Z)
• Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data [0.08192907805418582]
  This paper proposes a late fusion deep learning model (LF-DLM) for semantic segmentation. One branch integrates detailed textures from aerial imagery captured by UNetFormer with a Multi-Axis Vision Transformer (ViT) backbone. The other branch captures complex-temporal dynamics from the Sentinel-2 satellite imageMax time series using a U-ViNet with Temporal Attention (U-TAE)
  arXiv  Detail & Related papers  (2024-10-01T07:50:37Z)
• Homography Guided Temporal Fusion for Road Line and Marking Segmentation [73.47092021519245]
  Road lines and markings are frequently occluded in the presence of moving vehicles, shadow, and glare. We propose a Homography Guided Fusion (HomoFusion) module to exploit temporally-adjacent video frames for complementary cues. We show that exploiting available camera intrinsic data and ground plane assumption for cross-frame correspondence can lead to a light-weight network with significantly improved performances in speed and accuracy.
  arXiv  Detail & Related papers  (2024-04-11T10:26:40Z)
• View Consistent Purification for Accurate Cross-View Localization [59.48131378244399]
  This paper proposes a fine-grained self-localization method for outdoor robotics. The proposed method addresses limitations in existing cross-view localization methods. It is the first sparse visual-only method that enhances perception in dynamic environments.
  arXiv  Detail & Related papers  (2023-08-16T02:51:52Z)
• Boosting 3-DoF Ground-to-Satellite Camera Localization Accuracy via Geometry-Guided Cross-View Transformer [66.82008165644892]
  We propose a method to increase the accuracy of a ground camera's location and orientation by estimating the relative rotation and translation between the ground-level image and its matched/retrieved satellite image. Experimental results demonstrate that our method significantly outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2023-07-16T11:52:27Z)
• 4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions [54.59279160621111]
  We present a novel visual SLAM and long-term localization benchmark for autonomous driving in challenging conditions based on the large-scale 4Seasons dataset. The proposed benchmark provides drastic appearance variations caused by seasonal changes and diverse weather and illumination conditions. We introduce a new unified benchmark for jointly evaluating visual odometry, global place recognition, and map-based visual localization performance.
  arXiv  Detail & Related papers  (2022-12-31T13:52:36Z)
• Monocular BEV Perception of Road Scenes via Front-to-Top View Projection [57.19891435386843]
  We present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view. Our model runs at 25 FPS on a single GPU, which is efficient and applicable for real-time panorama HD map reconstruction.
  arXiv  Detail & Related papers  (2022-11-15T13:52:41Z)
• Uncertainty-aware Perception Models for Off-road Autonomous Unmanned Ground Vehicles [6.2574402913714575]
  Off-road autonomous unmanned ground vehicles (UGVs) are being developed for military and commercial use to deliver crucial supplies in remote locations. Current datasets used to train perception models for off-road autonomous navigation lack of diversity in seasons, locations, semantic classes, as well as time of day. We investigate how to combine multiple datasets to train a semantic segmentation-based environment perception model. We show that training the model to capture uncertainty could improve the model performance by a significant margin.
  arXiv  Detail & Related papers  (2022-09-22T15:59:33Z)
• Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image [91.29546868637911]
  This paper addresses the problem of vehicle-mounted camera localization by matching a ground-level image with an overhead-view satellite map. The key idea is to formulate the task as pose estimation and solve it by neural-net based optimization. Experiments on standard autonomous vehicle localization datasets have confirmed the superiority of the proposed method.
  arXiv  Detail & Related papers  (2022-04-10T19:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.