DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding

• URL: http://arxiv.org/abs/2603.19219v1
• Date: Thu, 19 Mar 2026 17:58:22 GMT
• Title: DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding
• Abstract summary: DriveTok is an efficient 3D driving scene tokenizer for unified multi-view reconstruction and understanding.<n>For decoding, we employ a multi-view transformer to reconstruct multi-view features from the scene tokens.<n>We also add a 3D head directly on the scene tokens for 3D semantic occupancy prediction.
• Score: 75.41918200553525
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the growing adoption of vision-language-action models and world models in autonomous driving systems, scalable image tokenization becomes crucial as the interface for the visual modality. However, most existing tokenizers are designed for monocular and 2D scenes, leading to inefficiency and inter-view inconsistency when applied to high-resolution multi-view driving scenes. To address this, we propose DriveTok, an efficient 3D driving scene tokenizer for unified multi-view reconstruction and understanding. DriveTok first obtains semantically rich visual features from vision foundation models and then transforms them into the scene tokens with 3D deformable cross-attention. For decoding, we employ a multi-view transformer to reconstruct multi-view features from the scene tokens and use multiple heads to obtain RGB, depth, and semantic reconstructions. We also add a 3D head directly on the scene tokens for 3D semantic occupancy prediction for better spatial awareness. With the multiple training objectives, DriveTok learns unified scene tokens that integrate semantic, geometric, and textural information for efficient multi-view tokenization. Extensive experiments on the widely used nuScenes dataset demonstrate that the scene tokens from DriveTok perform well on image reconstruction, semantic segmentation, depth prediction, and 3D occupancy prediction tasks.

Related papers

• Geometry-Guided 3D Visual Token Pruning for Video-Language Models [35.786221225043455]
  We propose Geo3DPruner, a Geometry-Guided 3D Visual Token Pruning framework.<n>Geo3DPruner first models cross-frame relevance through geometry-aware global attention, and then performs a two-stage pruning process.<n>Experiments on multiple 3D scene understanding benchmarks demonstrate that Geo3DPruner retains over 90% of the original performance while pruning 90% of visual tokens, significantly outperforming existing text-guided and vision-guided pruning methods.
  arXiv  Detail & Related papers  (2026-04-20T13:33:50Z)
• Unified Semantic Transformer for 3D Scene Understanding [55.415468022487005]
  We introduce UNITE, a novel feed-forward neural network that unifies a diverse set of 3D semantic tasks within a single model.<n>Our model operates on unseen scenes in a fully end-to-end manner and only takes a few seconds to infer the full 3D semantic geometry.<n>We demonstrate that UNITE achieves state-of-the-art performance on several different semantic tasks and even outperforms task-specific models.
  arXiv  Detail & Related papers  (2025-12-16T12:49:35Z)
• Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving [54.85072592658933]
  We present Flex, an efficient and effective scene encoder that addresses the computational bottleneck of processing high-volume multi-camera data in autonomous driving.<n>By design, our approach is geometry-agnostic, learning a compact scene representation directly from data without relying on the explicit 3D inductive biases.<n>Our findings challenge the prevailing assumption that 3D priors are necessary, demonstrating that a data-driven, joint encoding strategy offers a more scalable, efficient and effective path for future autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding [39.18256367776712]
  We present NDTokenizer3D, a 3D VLM that performs a wide range of 3D scene understanding tasks while naturally supporting human interactions.<n>The core of our approach is a novel three-stage scene tokenization pipeline built upon a Multi-Scale Normal Distributions Transform (NDT) representation.<n>NDTokenizer3D first constructs a multi-scale NDT representation from raw high-resolution point clouds, preserving both global context and fine-grained geometric details.
  arXiv  Detail & Related papers  (2025-11-26T09:12:17Z)
• Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Driving [73.3702076688159]
  We propose a novel contrastive learning algorithm, Cohere3D, to learn coherent instance representations in a long-term input sequence. We evaluate our algorithm by finetuning the pretrained model on various downstream perception, prediction, and planning tasks.
  arXiv  Detail & Related papers  (2024-02-23T19:43:01Z)
• Neural Groundplans: Persistent Neural Scene Representations from a Single Image [90.04272671464238]
  We present a method to map 2D image observations of a scene to a persistent 3D scene representation. We propose conditional neural groundplans as persistent and memory-efficient scene representations.
  arXiv  Detail & Related papers  (2022-07-22T17:41:24Z)
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors [69.02332607843569]
  PriSMONet is a novel approach for learning Multi-Object 3D scene decomposition and representations from single images. A recurrent encoder regresses a latent representation of 3D shape, pose and texture of each object from an input RGB image. We evaluate the accuracy of our model in inferring 3D scene layout, demonstrate its generative capabilities, assess its generalization to real images, and point out benefits of the learned representation.
  arXiv  Detail & Related papers  (2020-10-08T14:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.