FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning

• URL: http://arxiv.org/abs/2507.23318v3
• Date: Tue, 16 Sep 2025 09:59:46 GMT
• Title: FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
• Abstract summary: We propose FastDriveVLA, a reconstruction-based vision token pruning framework for autonomous driving.<n>A novel foreground adversarial-background reconstruction strategy is designed to train ReconPruner for the visual encoder of VLA models.<n>Our approach achieves state-of-the-art results on the nuScenes open-loop planning benchmark across different pruning ratios.
• Score: 75.80110543049783
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-Language-Action (VLA) models have demonstrated significant potential in complex scene understanding and action reasoning, leading to their increasing adoption in end-to-end autonomous driving systems. However, the long visual tokens of VLA models greatly increase computational costs. Current visual token pruning methods in Vision-Language Models (VLM) rely on either visual token similarity or visual-text attention, but both have shown poor performance in autonomous driving scenarios. Given that human drivers concentrate on relevant foreground areas while driving, we assert that retaining visual tokens containing this foreground information is essential for effective decision-making. Inspired by this, we propose FastDriveVLA, a novel reconstruction-based vision token pruning framework designed specifically for autonomous driving. FastDriveVLA includes a plug-and-play visual token pruner called ReconPruner, which prioritizes foreground information through MAE-style pixel reconstruction. A novel adversarial foreground-background reconstruction strategy is designed to train ReconPruner for the visual encoder of VLA models. Once trained, ReconPruner can be seamlessly applied to different VLA models with the same visual encoder without retraining. To train ReconPruner, we also introduce a large-scale dataset called nuScenes-FG, consisting of 241K image-mask pairs with annotated foreground regions. Our approach achieves state-of-the-art results on the nuScenes open-loop planning benchmark across different pruning ratios.

Related papers

• DriveStack-VLA: Render-Teacher Alignment for BEV-Based DeepStack Vision-Language-Action Model [22.06522773748522]
  We present DriveStack-VLA, a framework built upon a large VLM backbone.<n>We develop dual visual modeling components to strengthen the spatial grounding of VLA driving.<n>DriveStack-VLA achieves 91.6 PDMS on NAVSIMv1, 91.0S on NAVSIMv2, and a driving score of 79.49 with a success rate of 56.36% on the closed-loop Bench2Drive.
  arXiv  Detail & Related papers  (2026-06-23T01:40:54Z)
• Grounding Driving VLA via Inverse Kinematics [32.65844881404964]
  We show that trajectory recovery requires both a current and a future visual state as boundary conditions.<n>We re-design Driving VLA in the style of an inverse kinematics solver.<n>With this simple prescription alone, our 0.5B-scale model recovers visual grounding and reaches trajectory planning performance comparable to 7B--8B VLAs.
  arXiv  Detail & Related papers  (2026-05-20T11:45:32Z)
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving [54.57163800903507]
  We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving.<n>A unified VLM backbone produces AR language tokens and flow-matching continuous action trajectories in a single forward pass over one shared representation.<n>On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time.
  arXiv  Detail & Related papers  (2026-05-12T18:09:42Z)
• ETA-VLA: Efficient Token Adaptation via Temporal Fusion and Intra-LLM Sparsification for Vision-Language-Action Models [14.841746872486333]
  We propose ETA-VLA, an Efficient Token Adaptation framework for VLA models.<n>We show that ETA-VLA achieves driving performance comparable to state-of-the-art baselines.<n>Our method prunes 85% of visual tokens and reduces inference FLOPs by 61%, but still retains 94% of the original accuracy on the NAVSIM v2 benchmark.
  arXiv  Detail & Related papers  (2026-03-26T06:53:42Z)
• Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models [66.96421290733126]
  Vision-Language-Action (VLA) models have emerged as a promising paradigm for robotic manipulation.<n>We propose textbfDeepVision-VLA, built on a textbfVision-Language Mixture-of-Transformers (VL-MoT) framework.<n>DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively.
  arXiv  Detail & Related papers  (2026-03-16T17:59:54Z)
• BFA++: Hierarchical Best-Feature-Aware Token Prune for Multi-View Vision Language Action Model [44.72361174037017]
  Vision-Language-Action (VLA) models have achieved significant breakthroughs by leveraging Large Vision Language Models (VLMs) to jointly interpret instructions and visual inputs.<n>The substantial increase in visual tokens, particularly from multi-view inputs, poses serious challenges to real-time robotic manipulation.<n>We propose BFA++, a dynamic token pruning framework designed specifically for VLA models.
  arXiv  Detail & Related papers  (2026-02-24T05:31:52Z)
• Visual Generation Tuning [84.50113837230333]
  We propose VGT, Visual Generation Tuning, a novel paradigm designed to stimulate the underlying capabilities of visual generation within vision language models.<n>In image reconstruction tasks, we achieve 26.67 PSNR and 0.50 rFID at a 28x compression ratio, outperforming specialized VAEs.<n>Our proposed VGT showcases significant scaling promise and is versatile for endowing any VLMs trained for multimodal understanding with the capabilities of visual generation.
  arXiv  Detail & Related papers  (2025-11-28T18:57:13Z)
• Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving [24.2108745917843]
  Vision-Language Models (VLMs) have emerged as a promising paradigm in autonomous driving.<n>VLMs offer a unified framework for perception, reasoning, and decision-making by jointly modeling visual inputs and natural language instructions.<n>We propose Prune2Drive, a plug-and-play visual token pruning framework for multi-view VLMs in autonomous driving.
  arXiv  Detail & Related papers  (2025-08-18T18:47:26Z)
• TinyDrive: Multiscale Visual Question Answering with Selective Token Routing for Autonomous Driving [10.439455144126617]
  TinyDrive is a lightweight VLM for multi-view VQA in driving scenarios.<n>Our model comprises two key components including a multiscale vision encoder and a dual-level prioritization mechanism for tokens and sequences.<n>TinyDrive is first evaluated on our custom-curated VQA dataset, and it is subsequently tested on the public DriveLM benchmark.
  arXiv  Detail & Related papers  (2025-05-21T14:19:24Z)
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models [89.44024245194315]
  We introduce a method that incorporates explicit visual chain-of-thought (CoT) reasoning into vision-language-action models (VLAs)<n>We introduce CoT-VLA, a state-of-the-art 7B VLA that can understand and generate visual and action tokens.<n>Our experimental results demonstrate that CoT-VLA achieves strong performance, outperforming the state-of-the-art VLA model by 17% in real-world manipulation tasks and 6% in simulation benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T22:23:04Z)
• CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention [4.9349065371630045]
  We propose Context Aware Scene Prediction Transformer (CASPFormer), which can perform multi-modal motion prediction from spatialized Bird-Eye-View (BEV) images. Our system can be integrated with any upstream perception module that is capable of generating BEV images. We evaluate our model on the nuScenes dataset and show that it reaches state-of-the-art across multiple metrics.
  arXiv  Detail & Related papers  (2024-09-26T12:37:22Z)
• Enhancing End-to-End Autonomous Driving with Latent World Model [78.22157677787239]
  We propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving.<n> LAW predicts future scene features based on current features and ego trajectories.<n>This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks.
  arXiv  Detail & Related papers  (2024-06-12T17:59:21Z)
• Street-View Image Generation from a Bird's-Eye View Layout [95.36869800896335]
  Bird's-Eye View (BEV) Perception has received increasing attention in recent years. Data-driven simulation for autonomous driving has been a focal point of recent research. We propose BEVGen, a conditional generative model that synthesizes realistic and spatially consistent surrounding images.
  arXiv  Detail & Related papers  (2023-01-11T18:39:34Z)
• Beyond the Field-of-View: Enhancing Scene Visibility and Perception with Clip-Recurrent Transformer [28.326852785609788]
  FlowLens architecture explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view. Experiments and user studies involving offline and online video inpainting, as well as beyondFo-V perception tasks, demonstrate that Flows achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-11-21T09:34:07Z)
• Monocular BEV Perception of Road Scenes via Front-to-Top View Projection [57.19891435386843]
  We present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view. Our model runs at 25 FPS on a single GPU, which is efficient and applicable for real-time panorama HD map reconstruction.
  arXiv  Detail & Related papers  (2022-11-15T13:52:41Z)
• Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images [128.881857704338]
  We study the problem of extracting a directed graph representing the local road network in BEV coordinates, from a single onboard camera image. We show that the method can be extended to detect dynamic objects on the BEV plane. We validate our approach against powerful baselines and show that our network achieves superior performance.
  arXiv  Detail & Related papers  (2021-10-05T12:40:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.