Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving

• URL: http://arxiv.org/abs/2511.19221v1
• Date: Mon, 24 Nov 2025 15:28:25 GMT
• Title: Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
• Authors: Jianhua Han, Meng Tian, Jiangtong Zhu, Fan He, Huixin Zhang, Sitong Guo, Dechang Zhu, Hao Tang, Pei Xu, Yuze Guo, Minzhe Niu, Haojie Zhu, Qichao Dong, Xuechao Yan, Siyuan Dong, Lu Hou, Qingqiu Huang, Xiaosong Jia, Hang Xu,
• Abstract summary: We introduce Percept-WAM, a perception-enhanced World-Awareness-Action Model that implicitly integrates 2D/3D scene understanding abilities within a single vision-language model (VLM)<n>We propose a grid-conditioned prediction mechanism for dense object perception, incorporating IoU-aware scoring and parallel autoregressive decoding, improving stability in long-tail, far-range, and small-object scenarios.<n>Experiments show that Percept-WAM matches or surpasses classical detectors and segmenters on downstream perception benchmarks, achieving 51.7/58.9 mAP on 2D detection and nuScenes BEV 3D detection
• Score: 48.512353531499286
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Autonomous driving heavily relies on accurate and robust spatial perception. Many failures arise from inaccuracies and instability, especially in long-tail scenarios and complex interactions. However, current vision-language models are weak at spatial grounding and understanding, and VLA systems built on them therefore show limited perception and localization ability. To address these challenges, we introduce Percept-WAM, a perception-enhanced World-Awareness-Action Model that is the first to implicitly integrate 2D/3D scene understanding abilities within a single vision-language model (VLM). Instead of relying on QA-style spatial reasoning, Percept-WAM unifies 2D/3D perception tasks into World-PV and World-BEV tokens, which encode both spatial coordinates and confidence. We propose a grid-conditioned prediction mechanism for dense object perception, incorporating IoU-aware scoring and parallel autoregressive decoding, improving stability in long-tail, far-range, and small-object scenarios. Additionally, Percept-WAM leverages pretrained VLM parameters to retain general intelligence (e.g., logical reasoning) and can output perception results and trajectory control outputs directly. Experiments show that Percept-WAM matches or surpasses classical detectors and segmenters on downstream perception benchmarks, achieving 51.7/58.9 mAP on COCO 2D detection and nuScenes BEV 3D detection. When integrated with trajectory decoders, it further improves planning performance on nuScenes and NAVSIM, e.g., surpassing DiffusionDrive by 2.1 in PMDS on NAVSIM. Qualitative results further highlight its strong open-vocabulary and long-tail generalization.

Related papers

• AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Models [42.57469056850227]
  Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic perception and control.<n>We propose a novel framework that integrates depth estimation into VLA models to enrich 3D feature representations.
  arXiv  Detail & Related papers  (2026-02-11T09:57:32Z)
• Spatial-aware Vision Language Model for Autonomous Driving [16.149511148218497]
  Vision-Language Models (VLMs) show significant promise for end-to-end autonomous driving by leveraging the common sense embedded in language models.<n>Current image-based methods struggle with accurate metric spatial reasoning and geometric inference, leading to unreliable driving policies.<n>We propose LVLDrive, a novel framework specifically designed to upgrade existing VLMs with robust 3D metric spatial understanding for autonomous driving.
  arXiv  Detail & Related papers  (2025-12-30T16:35:00Z)
• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving [38.21244888074097]
  SpaceDrive is a spatial-aware driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens.<n>We show that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive benchmark.
  arXiv  Detail & Related papers  (2025-12-11T14:59:07Z)
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning [69.36145467833498]
  We introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving.<n> evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems.
  arXiv  Detail & Related papers  (2025-12-04T05:05:41Z)
• Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regression [12.590536117486257]
  Existing Vision Language Models (VLMs) struggle to comprehend real-world 3D spatial intelligence.<n> GEODE augments main VLM with two specialized, plug-and-play modules.<n>The synergy of these modules allows our 1.5B parameter model to function as a high-level semantic dispatcher.
  arXiv  Detail & Related papers  (2025-11-14T12:42:07Z)
• From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors [54.84863164684646]
  Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders.<n>In this work, we introduce FALCON, a novel paradigm that injects rich 3D spatial tokens into the action head.
  arXiv  Detail & Related papers  (2025-10-20T11:26:45Z)
• Universal Camouflage Attack on Vision-Language Models for Autonomous Driving [67.34987318443761]
  Visual language modeling for automated driving is emerging as a promising research direction.<n>VLM-AD remains vulnerable to serious security threats from adversarial attacks.<n>We propose the first Universal Camouflage Attack framework for VLM-AD.
  arXiv  Detail & Related papers  (2025-09-24T14:52:01Z)
• OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision [31.929268076595122]
  OccVLA is a novel framework that integrates 3D occupancy representations into a unified multimodal reasoning process.<n>OccVLA achieves state-of-the-art results on the nuScenes benchmark for trajectory planning and demonstrates superior performance on 3D visual question-answering tasks.
  arXiv  Detail & Related papers  (2025-09-06T03:47:21Z)
• VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception [5.245213543721097]
  We propose VLM-3D, the first end-to-end framework that enables 3D geometric perception in autonomous driving scenarios.<n>VLM-3D incorporates Low-Rank Adaptation (LoRA) to efficiently adapt VLMs to driving tasks with minimal computational overhead.<n>We show that the proposed joint semantic-geometric loss in VLM-3D leads to a 12.8% improvement in perception accuracy.
  arXiv  Detail & Related papers  (2025-08-12T16:25:27Z)
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning [68.45848423501927]
  We propose a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning.<n>Our approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions.
  arXiv  Detail & Related papers  (2025-04-06T03:54:21Z)
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning [68.45848423501927]
  We propose a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning.<n>Our approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions.
  arXiv  Detail & Related papers  (2024-05-02T17:59:24Z)
• DuEqNet: Dual-Equivariance Network in Outdoor 3D Object Detection for Autonomous Driving [4.489333751818157]
  We propose DuEqNet, which first introduces the concept of equivariance into 3D object detection network. The dual-equivariant of our model can extract the equivariant features at both local and global levels. Our model presents higher accuracy on orientation and better prediction efficiency.
  arXiv  Detail & Related papers  (2023-02-27T08:30:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.