Towards Viewpoint-Robust End-to-End Autonomous Driving with 3D Foundation Model Priors

• URL: http://arxiv.org/abs/2604.00597v1
• Date: Wed, 01 Apr 2026 08:04:40 GMT
• Title: Towards Viewpoint-Robust End-to-End Autonomous Driving with 3D Foundation Model Priors
• Abstract summary: We investigate an augmentation-free approach that leverages geometric priors from a 3D foundation model.<n>The method injects per-pixel 3D positions derived from depth estimates as positional embeddings and fuses intermediate geometric features through cross-attention.
• Score: 11.0589601314909
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robust trajectory planning under camera viewpoint changes is important for scalable end-to-end autonomous driving. However, existing models often depend heavily on the camera viewpoints seen during training. We investigate an augmentation-free approach that leverages geometric priors from a 3D foundation model. The method injects per-pixel 3D positions derived from depth estimates as positional embeddings and fuses intermediate geometric features through cross-attention. Experiments on the VR-Drive camera viewpoint perturbation benchmark show reduced performance degradation under most perturbation conditions, with clear improvements under pitch and height perturbations. Gains under longitudinal translation are smaller, suggesting that more viewpoint-agnostic integration is needed for robustness to camera viewpoint changes.

Related papers

• OmniPoint: Universal Monocular Metric Pointcloud from Any Camera [79.24417597712802]
  We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors.<n>To overcome projection rigidity, OmniPoint abandons conventional planar depth regression.<n>It instead adopts a decoupled ray and distance representation alongside a decoupled training objective.
  arXiv  Detail & Related papers  (2026-09-08T19:45:58Z)
• GeoStream: Toward Precise Camera Controlled Streaming Video Generation [85.01823220801256]
  Existing approaches learn camera motion implicitly, leading to inaccurate control under out-of-distribution trajectories.<n>Explicit geometric conditioning improves controllability, but existing methods are non-autoregressive.<n>We propose GeoStream, a framework that enables precise metric-scale camera control in autoregressive streaming video generation.
  arXiv  Detail & Related papers  (2026-06-13T07:18:04Z)
• Geometry-Aware Rotary Position Embedding for Consistent Video World Model [48.914346802616414]
  ViewRope is a geometry-aware encoding that injects camera-ray directions directly into video transformer self-attention layers.<n>Geometry-Aware Frame-Sparse Attention exploits these geometric cues to selectively attend to relevant historical frames.<n>Our results demonstrate that ViewRope substantially improves long-term consistency while reducing computational costs.
  arXiv  Detail & Related papers  (2026-02-08T08:01:16Z)
• ViewMorpher3D: A 3D-aware Diffusion Framework for Multi-Camera Novel View Synthesis in Autonomous Driving [20.935790354765604]
  We introduce ViewMorpher3D, a multi-view image enhancement framework based on image diffusion models.<n>Unlike single-view approaches, ViewMorpher3D jointly processes a set of rendered views conditioned on camera poses, 3D geometric priors, and temporally adjacent or spatially overlapping reference views.<n>Our framework accommodates variable numbers of cameras and flexible reference/target view configurations, making it adaptable to diverse sensor setups.
  arXiv  Detail & Related papers  (2026-01-12T13:44:14Z)
• DriveVGGT: Visual Geometry Transformer for Autonomous Driving [50.5036123750788]
  DriveVGGT is a scale-aware 4D reconstruction framework specifically designed for autonomous driving data.<n>We propose a temporal Video Attention (TVA) module to process multi-camera videos independently.<n>Then, we propose a Multi-camera Consistency Attention (MCA) module to conduct window attention with normalized relative pose embeddings.
  arXiv  Detail & Related papers  (2025-11-27T09:40:43Z)
• FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views [100.45129752375658]
  We present FLARE, a feed-forward model designed to infer high-quality camera poses and 3D geometry from uncalibrated sparse-view images.<n>Our solution features a cascaded learning paradigm with camera pose serving as the critical bridge, recognizing its essential role in mapping 3D structures onto 2D image planes.
  arXiv  Detail & Related papers  (2025-02-17T18:54:05Z)
• UniDrive: Towards Universal Driving Perception Across Camera Configurations [38.40168936403638]
  3D perception aims to infer 3D information from 2D images based on 3D-2D projection.<n>Generalizing across camera configurations is important for deploying autonomous driving models on different car models.<n>We present UniDrive, a novel framework for vision-centric autonomous driving to achieve universal perception across camera configurations.
  arXiv  Detail & Related papers  (2024-10-17T17:59:59Z)
• Towards Model Generalization for Monocular 3D Object Detection [57.25828870799331]
  We present an effective unified camera-generalized paradigm (CGP) for Mono3D object detection. We also propose the 2D-3D geometry-consistent object scaling strategy (GCOS) to bridge the gap via an instance-level augment. Our method called DGMono3D achieves remarkable performance on all evaluated datasets and surpasses the SoTA unsupervised domain adaptation scheme.
  arXiv  Detail & Related papers  (2022-05-23T23:05:07Z)
• MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision [72.5863451123577]
  We show how to train a neural model that can perform accurate 3D pose and camera estimation. Our method outperforms both classical bundle adjustment and weakly-supervised monocular 3D baselines.
  arXiv  Detail & Related papers  (2021-08-10T18:39:56Z)
• Wide-Baseline Relative Camera Pose Estimation with Directional Learning [46.21836501895394]
  We introduce DirectionNet, which estimates discrete distributions over the 5D relative pose space using a novel parameterization to make the estimation problem tractable. We evaluate our model on challenging synthetic and real pose estimation datasets constructed from Matterport3D and InteriorNet.
  arXiv  Detail & Related papers  (2021-06-07T04:46:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.