InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving

• URL: http://arxiv.org/abs/2503.13047v1
• Date: Mon, 17 Mar 2025 10:52:32 GMT
• Title: InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving
• Authors: Ruiqi Song, Xianda Guo, Hangbin Wu, Qinggong Wei, Long Chen,
• Abstract summary: We propose a novel end-to-end autonomous driving method called InsightDrive.<n>It organizes perception by language-guided scene representation.<n>In experiments, InsightDrive achieves state-of-the-art performance in end-to-end autonomous driving.
• Score: 3.8737986316149775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Directly generating planning results from raw sensors has become increasingly prevalent due to its adaptability and robustness in complex scenarios. Scene representation, as a key module in the pipeline, has traditionally relied on conventional perception, which focus on the global scene. However, in driving scenarios, human drivers typically focus only on regions that directly impact driving, which often coincide with those required for end-to-end autonomous driving. In this paper, a novel end-to-end autonomous driving method called InsightDrive is proposed, which organizes perception by language-guided scene representation. We introduce an instance-centric scene tokenizer that transforms the surrounding environment into map- and object-aware instance tokens. Scene attention language descriptions, which highlight key regions and obstacles affecting the ego vehicle's movement, are generated by a vision-language model that leverages the cognitive reasoning capabilities of foundation models. We then align scene descriptions with visual features using the vision-language model, guiding visual attention through these descriptions to give effectively scene representation. Furthermore, we employ self-attention and cross-attention mechanisms to model the ego-agents and ego-map relationships to comprehensively build the topological relationships of the scene. Finally, based on scene understanding, we jointly perform motion prediction and planning. Extensive experiments on the widely used nuScenes benchmark demonstrate that the proposed InsightDrive achieves state-of-the-art performance in end-to-end autonomous driving. The code is available at https://github.com/songruiqi/InsightDrive

Related papers

• Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning [24.511628941825116]
  We introduce Sce2DriveX, a human-like driving chain-of-thought (CoT) reasoning framework framework.<n>It reconstructs the implicit cognitive chain inherent in human driving, covering scene understanding, meta-action reasoning, behavior interpretation analysis, motion planning and control.<n>It achieves state-of-the-art performance from scene understanding to end-to-end driving, as well as robust generalization on the CARLA Bench2Drive benchmark.
  arXiv  Detail & Related papers  (2025-02-19T09:50:44Z)
• Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene [56.73568220959019]
  Collaborative autonomous driving (CAV) seems like a promising direction, but collecting data for development is non-trivial. We introduce a novel surrogate to the rescue, which is to generate realistic perception from different viewpoints in a driving scene. We present the very first solution, using a combination of simulated collaborative data and real ego-car data.
  arXiv  Detail & Related papers  (2025-02-10T17:07:53Z)
• Doe-1: Closed-Loop Autonomous Driving with Large World Model [63.99937807085461]
  We propose a large Driving wOrld modEl (Doe-1) for unified perception, prediction, and planning.<n>We use free-form texts for perception and generate future predictions directly in the RGB space with image tokens.<n>For planning, we employ a position-aware tokenizer to effectively encode action into discrete tokens.
  arXiv  Detail & Related papers  (2024-12-12T18:59:59Z)
• Enhancing End-to-End Autonomous Driving with Latent World Model [78.22157677787239]
  We propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving.<n> LAW predicts future scene features based on current features and ego trajectories.<n>This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks.
  arXiv  Detail & Related papers  (2024-06-12T17:59:21Z)
• Embodied Understanding of Driving Scenarios [44.21311841582762]
  Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Here, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities.
  arXiv  Detail & Related papers  (2024-03-07T15:39:18Z)
• GenAD: Generative End-to-End Autonomous Driving [13.332272121018285]
  GenAD is a generative framework that casts autonomous driving into a generative modeling problem. We propose an instance-centric scene tokenizer that first transforms the surrounding scenes into map-aware instance tokens. We then employ a variational autoencoder to learn the future trajectory distribution in a structural latent space for trajectory prior modeling.
  arXiv  Detail & Related papers  (2024-02-18T08:21:05Z)
• On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving [37.617793990547625]
  This report provides an exhaustive evaluation of the latest state-of-the-art VLM, GPT-4V. We explore the model's abilities to understand and reason about driving scenes, make decisions, and ultimately act in the capacity of a driver. Our findings reveal that GPT-4V demonstrates superior performance in scene understanding and causal reasoning compared to existing autonomous systems.
  arXiv  Detail & Related papers  (2023-11-09T12:58:37Z)
• DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving [76.24483706445298]
  We introduce DriveDreamer, a world model entirely derived from real-world driving scenarios. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.
  arXiv  Detail & Related papers  (2023-09-18T13:58:42Z)
• ADAPT: Action-aware Driving Caption Transformer [24.3857045947027]
  We propose an end-to-end transformer-based architecture, ADAPT, which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. Experiments on BDD-X dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time.
  arXiv  Detail & Related papers  (2023-02-01T18:59:19Z)
• Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving [58.879758550901364]
  Recent perception systems enhance spatial understanding with sensor fusion but often lack full environmental context. We introduce a framework that integrates three cameras to emulate the human field of view, coupled with top-down bird-eye-view semantic data to enhance contextual representation. Our method achieves displacement error by 0.67m in open-loop settings, surpassing current methods by 6.9% on the nuScenes dataset.
  arXiv  Detail & Related papers  (2022-10-13T05:56:20Z)
• SceneGen: Learning to Generate Realistic Traffic Scenes [92.98412203941912]
  We present SceneGen, a neural autoregressive model of traffic scenes that eschews the need for rules and distributions. We demonstrate SceneGen's ability to faithfully model distributions of real traffic scenes.
  arXiv  Detail & Related papers  (2021-01-16T22:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.