Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception

• URL: http://arxiv.org/abs/2503.13587v1
• Date: Mon, 17 Mar 2025 17:59:50 GMT
• Title: Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception
• Authors: Dingkang Liang, Dingyuan Zhang, Xin Zhou, Sifan Tu, Tianrui Feng, Xiaofan Li, Yumeng Zhang, Mingyang Du, Xiao Tan, Xiang Bai,
• Abstract summary: We present UniFuture, a driving world model that seamlessly integrates future scene generation and perception within a single framework.<n>Our approach jointly models future appearance (i.e., RGB image) and geometry (i.e., depth), ensuring coherent predictions.
• Score: 47.65526944865586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present UniFuture, a simple yet effective driving world model that seamlessly integrates future scene generation and perception within a single framework. Unlike existing models focusing solely on pixel-level future prediction or geometric reasoning, our approach jointly models future appearance (i.e., RGB image) and geometry (i.e., depth), ensuring coherent predictions. Specifically, during the training, we first introduce a Dual-Latent Sharing scheme, which transfers image and depth sequence in a shared latent space, allowing both modalities to benefit from shared feature learning. Additionally, we propose a Multi-scale Latent Interaction mechanism, which facilitates bidirectional refinement between image and depth features at multiple spatial scales, effectively enhancing geometry consistency and perceptual alignment. During testing, our UniFuture can easily predict high-consistency future image-depth pairs by only using the current image as input. Extensive experiments on the nuScenes dataset demonstrate that UniFuture outperforms specialized models on future generation and perception tasks, highlighting the advantages of a unified, structurally-aware world model. The project page is at https://github.com/dk-liang/UniFuture.

Related papers

• Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction [60.964512894143475]
  We present Generative Spatial Transformer ( GST), a novel auto-regressive framework that jointly addresses spatial localization and view prediction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction.
  arXiv  Detail & Related papers  (2024-10-24T17:58:05Z)
• Restyling Unsupervised Concept Based Interpretable Networks with Generative Models [14.604305230535026]
  We propose a novel method that relies on mapping the concept features to the latent space of a pretrained generative model. We quantitatively ascertain the efficacy of our method in terms of accuracy of the interpretable prediction network, fidelity of reconstruction, as well as faithfulness and consistency of learnt concepts.
  arXiv  Detail & Related papers  (2024-07-01T14:39:41Z)
• Visual Representation Learning with Stochastic Frame Prediction [90.99577838303297]
  This paper revisits the idea of video generation that learns to capture uncertainty in frame prediction. We design a framework that trains a frame prediction model to learn temporal information between frames. We find this architecture allows for combining both objectives in a synergistic and compute-efficient manner.
  arXiv  Detail & Related papers  (2024-06-11T16:05:15Z)
• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)
• Breathing New Life into 3D Assets with Generative Repainting [74.80184575267106]
  Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, and orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools.
  arXiv  Detail & Related papers  (2023-09-15T16:34:51Z)
• ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning [132.20119288212376]
  We propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously. To the best of our knowledge, we are the first to systematically investigate each part of an interpretable end-to-end vision-based autonomous driving system.
  arXiv  Detail & Related papers  (2022-07-15T16:57:43Z)
• Unified Recurrence Modeling for Video Action Anticipation [16.240254363118016]
  We propose a unified recurrence modeling for video action anticipation via message passing framework. Our proposed method outperforms previous works on the large-scale EPIC-Kitchen dataset.
  arXiv  Detail & Related papers  (2022-06-02T12:16:44Z)
• Simple Video Generation using Neural ODEs [9.303957136142293]
  We learn latent variable models that predict the future in latent space and project back to pixels. We show that our approach yields promising results in the task of future frame prediction on the Moving MNIST dataset with 1 and 2 digits.
  arXiv  Detail & Related papers  (2021-09-07T19:03:33Z)
• Future Frame Prediction of a Video Sequence [5.660207256468971]
  The ability to predict, anticipate and reason about future events is the essence of intelligence. The ability to predict, anticipate and reason about future events is the essence of intelligence.
  arXiv  Detail & Related papers  (2020-08-31T15:31:02Z)
• Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View with a Reachability Prior [36.80686175878314]
  We investigate the problem of anticipating future dynamics, particularly the future location of other vehicles and pedestrians, in the view of a moving vehicle. We estimate a reachability prior for certain classes of objects from the semantic map of the present image and propagate it into the future using the planned egomotion. Experiments show that the reachability prior combined with multi-hypotheses learning improves multimodal prediction of the future location of tracked objects and, for the first time, the emergence of new objects.
  arXiv  Detail & Related papers  (2020-06-08T15:57:26Z)
• Probabilistic Future Prediction for Video Scene Understanding [11.236856606065514]
  We present a novel deep learning architecture for probabilistic future prediction from video. We predict the future semantics, motion of complex real-world urban scenes and use this representation to control an autonomous vehicle.
  arXiv  Detail & Related papers  (2020-03-13T17:48:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.