Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Driving

• URL: http://arxiv.org/abs/2608.17178v1
• Date: Mon, 17 Aug 2026 22:39:49 GMT
• Title: Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Driving
• Abstract summary: We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy.<n>V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on City, and 3.75 RMSE on KITTI-2015 depth, while incurring only 14% additional pre-training overhead.
• Score: 57.56568641990586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video self-supervised learning through masked spatiotemporal prediction has emerged as a promising paradigm for learning feature representations from unlabeled data. However, existing methods typically rely on random masking, which indiscriminately removes regions irrespective of their semantic or temporal relevance. In ego-centric driving videos, this can weaken the pretext signal since safety-critical cues such as pedestrians, vehicles, lane boundaries, and dynamic interactions often occupy only a small portion of the frame, yet are central to downstream perception. We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy. It accounts for semantically and temporally relevant context. The proposed policy preserves and predicts context according to semantic importance and temporal relevance, yielding more informative representation learning while retaining the efficiency of masked prediction. We evaluate the resulting encoders on four driving benchmarks spanning tracking, semantic segmentation, and depth estimation. The results demonstrate that V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on Cityscapes, and 3.75 RMSE on KITTI-2015 depth, while incurring only ~14% additional pre-training iteration overhead.

Related papers

• DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving [89.17723223258513]
  Video panoptic segmentation (DVPS) unifies metric depth, semantic segmentation, and instance trajectories.<n>Existing approaches often rely on computationally expensive, multi-stage pipelines or offline tracking, rendering them unsuitable for real-time decision-making.<n>We propose DVPSFormer, a unified online architecture designed for efficient 4D scene understanding.
  arXiv  Detail & Related papers  (2026-07-28T18:16:00Z)
• TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Driving [1.5327485092552822]
  Birds-Eye View (BEV) end-to-end instance prediction has emerged as a robust paradigm for autonomous driving perception.<n>Current state-of-the-art approaches rely predominantly on geometric supervision, such as occupancy regression and optical flow.<n>We introduce Text-Guided Representation for Instance Prediction (TGRIP), a novel framework that bridges this gap by injecting rich semantic priors into the instance prediction loop.
  arXiv  Detail & Related papers  (2026-07-06T08:47:56Z)
• How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study [2.6954666679827137]
  VENUSS is a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes.<n>It extracts temporal sequences from driving videos, and generates structured evaluations across custom categories.<n>We show how even top models achieve only 57% accuracy, not matching human performance in similar constraints.
  arXiv  Detail & Related papers  (2026-04-08T07:14:55Z)
• Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving [16.94669292450282]
  This paper proposes FM4SU, a novel methodology for training a symbolic foundation model (FM) for scene understanding in autonomous driving.<n>It leverages knowledge graphs (KGs) to capture sensory observation along with domain knowledge such as road topology, traffic rules, or complex interactions between traffic participants.<n>The results demonstrate that fine-tuned models achieve significantly higher accuracy in all tasks.
  arXiv  Detail & Related papers  (2025-03-24T14:38:25Z)
• DAVE: Diverse Atomic Visual Elements Dataset with High Representation of Vulnerable Road Users in Complex and Unpredictable Environments [60.69159598130235]
  We present a new dataset, DAVE, designed for evaluating perception methods with high representation of Vulnerable Road Users (VRUs)<n>DAVE is a manually annotated dataset encompassing 16 diverse actor categories (spanning animals, humans, vehicles, etc.) and 16 action types (complex and rare cases like cut-ins, zigzag movement, U-turn, etc.)<n>Our experiments show that existing methods suffer degradation in performance when evaluated on DAVE, highlighting its benefit for future video recognition research.
  arXiv  Detail & Related papers  (2024-12-28T06:13:44Z)
• GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction [12.613528624623514]
  This research is the first to conduct both quantitative and qualitative evaluations of Vision Language Models (VLMs) in the context of pedestrian behavior prediction for autonomous driving. We evaluate GPT-4V on publicly available pedestrian datasets: JAAD and WiDEVIEW. The model achieves a 57% accuracy in a zero-shot manner, which, while impressive, is still behind the state-of-the-art domain-specific models (70%) in predicting pedestrian crossing actions.
  arXiv  Detail & Related papers  (2023-11-24T18:02:49Z)
• Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving [68.95178518732965]
  A self-driving vehicle (SDV) must be able to perceive its surroundings and predict the future behavior of other traffic participants. Existing works either perform object detection followed by trajectory of the detected objects, or predict dense occupancy and flow grids for the whole scene. This motivates our unified approach to perception and future prediction that implicitly represents occupancy and flow over time with a single neural network.
  arXiv  Detail & Related papers  (2023-08-02T23:39:24Z)
• Policy Pre-training for End-to-end Autonomous Driving via Self-supervised Geometric Modeling [96.31941517446859]
  We propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
  arXiv  Detail & Related papers  (2023-01-03T08:52:49Z)
• Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention [0.0]
  Pedestrian crossing intention should be recognized in real-time for urban driving. Recent works have shown the potential of using vision-based deep neural network models for this task. This work introduces a neural network architecture to fuse inherently different novel-temporal features for pedestrian crossing intention prediction.
  arXiv  Detail & Related papers  (2021-04-12T14:10:25Z)
• Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting [91.69900691029908]
  We advocate for predicting both the individual motions as well as the scene occupancy map. We propose a Scene-Actor Graph Neural Network (SA-GNN) which preserves the relative spatial information of pedestrians. On two large-scale real-world datasets, we showcase that our scene-occupancy predictions are more accurate and better calibrated than those from state-of-the-art motion forecasting methods.
  arXiv  Detail & Related papers  (2021-01-07T06:08:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.