DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving

• URL: http://arxiv.org/abs/2503.07656v1
• Date: Fri, 07 Mar 2025 11:41:18 GMT
• Title: DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving
• Authors: Xiaosong Jia, Junqi You, Zhiyuan Zhang, Junchi Yan,
• Abstract summary: DriveTransformer is a simplified E2E-AD framework for the ease of scaling up.<n>It is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention.<n>It achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.
• Score: 62.62464518137153
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: End-to-end autonomous driving (E2E-AD) has emerged as a trend in the field of autonomous driving, promising a data-driven, scalable approach to system design. However, existing E2E-AD methods usually adopt the sequential paradigm of perception-prediction-planning, which leads to cumulative errors and training instability. The manual ordering of tasks also limits the system`s ability to leverage synergies between tasks (for example, planning-aware perception and game-theoretic interactive prediction and planning). Moreover, the dense BEV representation adopted by existing methods brings computational challenges for long-range perception and long-term temporal fusion. To address these challenges, we present DriveTransformer, a simplified E2E-AD framework for the ease of scaling up, characterized by three key features: Task Parallelism (All agent, map, and planning queries direct interact with each other at each block), Sparse Representation (Task queries direct interact with raw sensor features), and Streaming Processing (Task queries are stored and passed as history information). As a result, the new framework is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention, which significantly reduces the complexity of system and leads to better training stability. DriveTransformer achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.

Related papers

• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [56.424032454461695]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences. Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations. Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Driving [10.984203470464687]
  Vision-language models (VLMs) often suffer from limitations such as inadequate spatial perception and hallucination. We propose a retrieval-augmented decision-making (RAD) framework to enhance VLMs' capabilities to reliably generate meta-actions in autonomous driving scenes. We fine-tune VLMs on a dataset derived from the NuScenes dataset to enhance their spatial perception and bird's-eye view image comprehension capabilities.
  arXiv  Detail & Related papers  (2025-03-18T03:25:57Z)
• DiffAD: A Unified Diffusion Modeling Approach for Autonomous Driving [17.939192289319056]
  We introduce DiffAD, a novel diffusion probabilistic model that redefines autonomous driving as a conditional image generation task. Byizing heterogeneous targets onto a unified bird's-eye view (BEV) and modeling their latent distribution, DiffAD unifies various driving objectives. The reverse process iteratively refines the generated BEV image, resulting in more robust and realistic driving behaviors.
  arXiv  Detail & Related papers  (2025-03-15T15:23:35Z)
• HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder [3.0989923815412204]
  We propose a novel end-to-end autonomous driving framework, termed HiP-AD.<n>HiP-AD simultaneously performs perception, prediction, and planning within a unified decoder.<n>Experiments demonstrate that HiP-AD outperforms all existing end-to-end autonomous driving methods on the closed-loop benchmark Bench2Drive.
  arXiv  Detail & Related papers  (2025-03-11T16:52:45Z)
• FASIONAD : FAst and Slow FusION Thinking Systems for Human-Like Autonomous Driving with Adaptive Feedback [15.805379735361862]
  This paper presents FASIONAD, a novel dual-system framework inspired by the cognitive model "Thinking, Fast and Slow"<n>The fast system handles routine navigation tasks using rapid, data-driven path planning, while the slow system focuses on complex reasoning and decision-making in challenging or unfamiliar situations.<n>Visual prompts generated by the fast system enable human-like reasoning in the slow system, which provides high-quality feedback to enhance the fast system's decision-making.
  arXiv  Detail & Related papers  (2024-11-27T03:14:16Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• RepVF: A Unified Vector Fields Representation for Multi-task 3D Perception [64.80760846124858]
  This paper proposes a novel unified representation, RepVF, which harmonizes the representation of various perception tasks. RepVF characterizes the structure of different targets in the scene through a vector field, enabling a single-head, multi-task learning model. Building upon RepVF, we introduce RFTR, a network designed to exploit the inherent connections between different tasks.
  arXiv  Detail & Related papers  (2024-07-15T16:25:07Z)
• Multi-task Learning for Real-time Autonomous Driving Leveraging Task-adaptive Attention Generator [15.94714567272497]
  We present a new real-time multi-task network adept at three vital autonomous driving tasks: monocular 3D object detection, semantic segmentation, and dense depth estimation. To counter the challenge of negative transfer, which is the prevalent issue in multi-task learning, we introduce a task-adaptive attention generator. Our rigorously optimized network, when tested on the Cityscapes-3D datasets, consistently outperforms various baseline models.
  arXiv  Detail & Related papers  (2024-03-06T05:04:40Z)
• IntentNet: Learning to Predict Intention from Raw Sensor Data [86.74403297781039]
  In this paper, we develop a one-stage detector and forecaster that exploits both 3D point clouds produced by a LiDAR sensor as well as dynamic maps of the environment. Our multi-task model achieves better accuracy than the respective separate modules while saving computation, which is critical to reducing reaction time in self-driving applications.
  arXiv  Detail & Related papers  (2021-01-20T00:31:52Z)
• LiDAR-based Panoptic Segmentation via Dynamic Shifting Network [56.71765153629892]
  LiDAR-based panoptic segmentation aims to parse both objects and scenes in a unified manner. We propose the Dynamic Shifting Network (DS-Net), which serves as an effective panoptic segmentation framework in the point cloud realm. Our proposed DS-Net achieves superior accuracies over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-24T08:44:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.