Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

• URL: http://arxiv.org/abs/2605.23163v2
• Date: Mon, 25 May 2026 07:32:46 GMT
• Title: Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
• Abstract summary: We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units.<n>We show that Fast-dDrive redefines the speed-accuracy frontier for driving agents.
• Score: 54.31800246594724
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking $N$ stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to $0.32$m (a $22\%$ improvement). When integrated with SGLang, our framework delivers $12\times$ throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.

Related papers

• Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference [40.7778980764007]
  Vision--action (VLA) models have emerged as a key component in embodied AI.<n>We present Deltoris, an algorithm-hardware co-design framework for efficient diffusion-based VLA inference.
  arXiv  Detail & Related papers  (2026-08-05T04:17:03Z)
• WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA [27.102605117827398]
  Vision-Language-Action (VLA) models have emerged as a prominent paradigm for end-to-end autonomous driving.<n>They are constrained by high computational latency and exposure bias arising from sequential autoregressive decoding.<n>We introduce WAM-Diff2, a multi-task discrete diffusion VLA framework powered by a three-stage hierarchical strategy.
  arXiv  Detail & Related papers  (2026-08-02T06:45:16Z)
• PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Models [63.31550146180189]
  This paper introduces a training-free $mathbfP$ersistence-Aware $mathbfC$ompression and $mathbfA$ggregation (PCA) method to preserve high-sampling raw visual information before the encoding stage.<n>PCA consistently outperforms existing state-of-the-art approaches in both efficiency and accuracy.
  arXiv  Detail & Related papers  (2026-07-22T05:17:38Z)
• TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies [58.40033352838586]
  Existing Vision-Language-Action models (VLAs) only inherit a single fixed speed from training demonstrations.<n>We observe that the magnitude of each predicted action already governs how fast the robot moves.<n>We turn this observation into TempoVLA, a single VLA whose execution speed is controlled by an explicit condition.
  arXiv  Detail & Related papers  (2026-06-04T17:59:40Z)
• BlockVLA: Accelerating Autoregressive VLA via Block Diffusion Finetuning [41.5997751218601]
  BlockVLA is a framework that adapts pretrained AR backbones into an efficient discrete diffusion policy.<n>We conduct extensive evaluations on the LIBERO and SimplerEnv benchmarks.<n>Our model exhibits superior training efficiency, with success rates converging substantially faster than baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:37:51Z)
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving [54.57163800903507]
  We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving.<n>A unified VLM backbone produces AR language tokens and flow-matching continuous action trajectories in a single forward pass over one shared representation.<n>On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time.
  arXiv  Detail & Related papers  (2026-05-12T18:09:42Z)
• Enabling Dynamic Tracking in Vision-Language-Action Models via Time-Discrete and Time-Continuous Velocity Feedforward [11.066720921275648]
  Vision-language-action (VLA) models have shown great promise for robot manipulation.<n>Their deployment on rigid industrial robots remains challenging due to the inherent trade-off between compliance and responsiveness.<n>This paper shows the importance of integrating velocity feedforward terms into VLA policies to resolve this trade-off.
  arXiv  Detail & Related papers  (2026-03-17T07:50:00Z)
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving [65.7087560656003]
  Generative diffusion models for end-to-end autonomous driving often suffer from mode collapse.<n>We propose DiffusionDriveV2, which leverages reinforcement learning to constrain low-quality modes and explore for superior trajectories.<n>This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
  arXiv  Detail & Related papers  (2025-12-08T17:29:52Z)
• SpecVLM: Fast Speculative Decoding in Vision-Language Models [14.243294546325714]
  Speculative decoding is a powerful way to accelerate autoregressive large language models (LLMs)<n>We study speculative decoding for vision-language models (VLMs)<n>We introduce SpecVLM, a practical system that delivers 1.5--2.3x end-to-end speedups over full autoregressive inference.
  arXiv  Detail & Related papers  (2025-09-15T11:53:56Z)
• SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration [70.72227437717467]
  Vision-Language-Action (VLA) models have attracted increasing attention for their strong control capabilities.<n>Their high computational cost and low execution frequency hinder their suitability for real-time tasks such as robotic manipulation and autonomous navigation.<n>We propose SP-VLA, a unified framework that accelerates VLA models by jointly scheduling models and pruning tokens.
  arXiv  Detail & Related papers  (2025-06-15T05:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.