VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving

• URL: http://arxiv.org/abs/2606.12396v1
• Date: Wed, 10 Jun 2026 17:57:06 GMT
• Title: VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving
• Abstract summary: Vision-language-action models can describe scenes and reason about them in language, but struggle to ground their actions in the dense 3D world around them.<n>We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through.
• Score: 6.130994037021129
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50\,m average) and 3-second collision rate (0.18\%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.

Related papers

• Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation [76.16065370700488]
  We introduce Lift3D-VLA, a unified VLA framework that equips models with explicit 3D point cloud reasoning and enables temporally coherent action generation.<n>We show that Lift3D-VLA achieves 10.8% and 11.1% higher mean success rates on MetaWorld and RLBench than the best-performing prior VLA methods.
  arXiv  Detail & Related papers  (2026-07-07T17:59:47Z)
• GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation [49.16739604572808]
  Vision-Language-Action (VLA) models achieve strong benchmark performance but struggle in real-world deployment with unseen objects.<n>We argue that this stems from the lack of a unified geometry-aware manipulation representation.<n>We propose GEAR-VLA, a VLA framework for learning unified geometry-aware action representations for generalizable robotic manipulation.
  arXiv  Detail & Related papers  (2026-06-07T09:23:16Z)
• PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction [57.63073414949329]
  We propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process.<n>PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-attention, enabling evolving action tokens to densely attend to both local geometric detail and global scene structure.
  arXiv  Detail & Related papers  (2026-05-20T17:10:31Z)
• VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models [55.12929235609365]
  Visual backbones of current vision-language-action (VLA) models are predominantly pretrained on 2D image data without explicit 3D geometric supervision.<n>Existing implicit grounding methods partially address this by aligning VLA features with those of 3D-aware foundation models.<n>We propose VEGA, a framework that directly aligns the output of the VLA's visual encoder with spatially-aware features from DINOv2-FiT3D.
  arXiv  Detail & Related papers  (2026-05-11T12:44:26Z)
• StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving [8.903468887918754]
  StyleVLA is a physics-informed VLA framework for generating diverse and physically plausible driving behaviors.<n>We construct a large-scale instruction dataset with over 1.2k scenarios, 76k Bird's Eye View (BEV) samples, and 42k First Person View (FPV) samples.<n>Experiments show that StyleVLA significantly outperforms proprietary models and state-of-the-art VLA models.
  arXiv  Detail & Related papers  (2026-03-10T10:33:58Z)
• IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance [35.90972175891154]
  Many Vision-Language-Action (VLA) models flatten image patches into a 1D token sequence, weakening the 2D spatial cues needed for precise manipulation.<n>We introduce a lightweight, training-free method that improves spatial understanding by exploiting affinity hints already available in the model's built-in vision.
  arXiv  Detail & Related papers  (2026-01-22T18:57:13Z)
• DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning [94.62097655403683]
  We propose DrivePI, a novel spatial-aware 4D MLLM that serves as a unified Vision-Language-Action framework.<n>Our method jointly performs spatial understanding, 3D perception, prediction (i.e., occupancy flow), and planning (i.e., action outputs) in parallel.<n>With only a 0.5B Qwen2.5 model as MLLM backbone, DrivePI as a single unified model matches or exceeds both existing VLA models and specialized VA models.
  arXiv  Detail & Related papers  (2025-12-14T18:45:54Z)
• D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation [66.7166217399105]
  Embodied agents face a critical dilemma that end-to-end models lack interpretability and explicit 3D reasoning.<n>Our model introduces two key innovations: 1) A Dynamic 3D Chain-of-Thought (3D CoT) that unifies planning, grounding, navigation, and question answering within a single 3D-VLM and CoT pipeline; 2) A Synergistic Learning from Fragmented Supervision (SLFS) strategy, which uses a masked autoregressive loss to learn from massive and partially-annotated hybrid data.
  arXiv  Detail & Related papers  (2025-12-14T09:53:15Z)
• LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning [63.19329995235114]
  Key bottleneck is that current scene representations struggle to balance performance and efficiency.<n>We propose the condensed feature grid (CFG), an efficient scene representation featuring significantly reduced token overhead and strong perception capability.<n>We introduce LEO-VL, a 3D VLM trained on 700k 3D-VL data spanning four real-world indoor domains and five tasks such as captioning and dialogue.
  arXiv  Detail & Related papers  (2025-06-11T16:56:34Z)
• PointVLA: Injecting the 3D World into Vision-Language-Action Models [10.758939578236582]
  We propose PointVLA, a framework that enhances pre-trained vision-language-action models with point cloud inputs without requiring retraining.<n>Our method freezes the vanilla action expert and injects 3D features via a lightweight modular block.<n>PointVLA outperforms state-of-the-art 2D imitation learning methods across both simulated and real-world robotic tasks.
  arXiv  Detail & Related papers  (2025-03-10T16:32:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.