Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis

• URL: http://arxiv.org/abs/2606.01590v1
• Date: Mon, 01 Jun 2026 02:37:56 GMT
• Title: Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis
• Abstract summary: We introduce StreetNVS, a video diffusion framework that conditions on all three signals through a Reference-Enhanced Camera Attention module.<n>We show capabilities of synthesizing coherent videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation.
• Score: 53.67632483195509
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern vehicle platforms are equipped with a rich sensor suite, including LiDAR, calibrated multi-camera rigs, and accurate ego-motion, that in principle offers strong signal for re-rendering a driving scene from novel viewpoints. A growing line of recent work leverages video diffusion models for this task, using their generative priors to synthesize plausible novel views from sparse vehicle observations. In practice, however, existing methods exploit only a fragment of this signal, and their quality tends to degrade as the target trajectory departs from the recorded driving path. We argue that this is fundamentally a multi-sensor fusion problem: sparse LiDAR reprojections supply accurate but incomplete metric geometry, surround-view reference imagery supplies dense appearance but no metric depth, and camera poses tie the two together across views. We introduce StreetNVS, a video diffusion framework that jointly conditions on all three signals through a Reference-Enhanced Camera Attention module based on a relative ray-level positional encoding. We develop a two-stage curriculum training strategy that gradually exposes the model to increasingly sparse LiDAR. On the Waymo Open Dataset, StreetNVS substantially outperforms state-of-the-art baselines under sparse LiDAR conditioning, matches methods that rely on 10-100 times denser point clouds. We further show capabilities of synthesizing coherent videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation. Our website: https://streetnvs.github.io

Related papers

• MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving [48.042009630881275]
  MM-TRELLIS is a multi-modal version of TRELLIS for in-the-wild 3D vehicle generation.<n>It integrates LiDAR and image sensors from autonomous driving datasets into native 3D generative models.<n>Our method outperforms existing methods in high-fidelity 3D vehicle generation.
  arXiv  Detail & Related papers  (2026-06-23T08:31:39Z)
• ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation [61.64162666162948]
  ReinDriveGen is a framework that enables full controllability over dynamic driving scenes.<n>Our approach constructs a dynamic 3D point cloud scene from multi-frame LiDAR data.<n>Our approach renders the edited scene into 2D condition images that guide a video diffusion model to synthesize realistic driving videos.
  arXiv  Detail & Related papers  (2026-04-01T16:48:20Z)
• TrajLoom: Dense Future Trajectory Generation from Video [17.78215938363174]
  We propose a framework that predicts future trajectories and trajectories from past video context.<n>Compared with state-of-the-art methods our approach extends the prediction horizon from 24 to 81 frames.<n>The predicted trajectories directly support downstream video generation and editing.
  arXiv  Detail & Related papers  (2026-03-23T22:10:58Z)
• DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos [23.967885144420613]
  We present DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos. Our method achieves state-of-the-art performance in novel view synthesis.
  arXiv  Detail & Related papers  (2024-05-27T23:38:10Z)
• LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes [73.65115834242866]
  Photorealistic simulation plays a crucial role in applications such as autonomous driving. However, reconstruction quality suffers on street scenes due to collinear camera motions and sparser samplings at higher speeds. We propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes.
  arXiv  Detail & Related papers  (2024-05-01T23:07:12Z)
• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels [50.40632021583213]
  We propose synthesizing additional LiDAR point clouds from novel viewpoints without physically driving at dangerous positions. We train a deep learning model, which takes a LiDAR scan as input and predicts the future trajectory as output. A waypoint controller is then applied to this predicted trajectory to determine the throttle and steering labels of the ego-vehicle.
  arXiv  Detail & Related papers  (2023-08-02T20:46:43Z)
• Multi-modal Streaming 3D Object Detection [20.01800869678355]
  We propose an innovative camera-LiDAR streaming 3D object detection framework. It uses camera images instead of past LiDAR slices to provide an up-to-date, dense, and wide context for streaming perception. Our method is shown to be robust to missing camera images, narrow LiDAR slices, and small camera-LiDAR miscalibration.
  arXiv  Detail & Related papers  (2022-09-12T00:30:52Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.