RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation

• URL: http://arxiv.org/abs/2606.14010v1
• Date: Fri, 12 Jun 2026 01:06:42 GMT
• Title: RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
• Abstract summary: We propose RT-VLA, a lightweight, distilled VLA model that transfers the driving and reasoning capabilities of the state-of-the-art SimLingo model into a compact student.<n> RT-VLA preserves language-based reasoning and supports post-hoc explanation through offline language analysis of safety-critical driving moments without adding latency to real-time control.<n>Compared to the SimLingo teacher, RT-VLA maintains competitive closed-loop driving and language reasoning performance while reducing inference time by 44.8X in vision-only mode and 7.9X in vision+language mode.
• Score: 11.501308555096605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models have shown strong potential for end-to-end autonomous driving by jointly modeling visual perception, language reasoning, explainability and action prediction. However, their large vision-language backbones and reasoning modules introduce substantial inference latency and thereby prevent their deployment in the unforgiving reality of the road networks. We propose RT-VLA, a lightweight, distilled VLA model that transfers the driving and reasoning capabilities of the state-of-the-art SimLingo model into a compact student through multi-level supervised distillation. RT-VLA preserves language-based reasoning and supports post-hoc explanation through offline language analysis of safety-critical driving moments without adding latency to real-time control. Compared to the SimLingo teacher, RT-VLA maintains competitive closed-loop driving and language reasoning performance while reducing inference time by 44.8X in vision-only mode and 7.9X in vision+language mode. These results suggest that supervised distillation is a practical approach for building real-time, explainable VLA-style autonomous driving models.

Related papers

• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving [54.57163800903507]
  We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving.<n>A unified VLM backbone produces AR language tokens and flow-matching continuous action trajectories in a single forward pass over one shared representation.<n>On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time.
  arXiv  Detail & Related papers  (2026-05-12T18:09:42Z)
• SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving [10.980525810871827]
  We propose SAMoE-VLA, a scene-adaptive Vision-Language-Action framework that conditions expert selection on structured scene representations.<n>Our key idea is to derive the MoE routing signal from bird's-eye-view (BEV) features that encapsulates traffic scene context.<n>To support temporally consistent reasoning across world-knowledge, perception, language, and action, we introduce a Conditional Cross-Modal Causal Attention mechanism.
  arXiv  Detail & Related papers  (2026-03-09T08:54:03Z)
• SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios [104.10555123175055]
  A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving.<n>We propose SteerVLA, which produces fine-grained language instructions that steer a vision-language-action driving policy.<n>We evaluate SteerVLA on a challenging closed-loop benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset.
  arXiv  Detail & Related papers  (2026-02-09T09:54:02Z)
• TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments [23.799083918923344]
  We introduce Think-in-Control (TIC)-VLA, a latency-aware framework that explicitly models delayed semantic reasoning during action generation.<n>TIC-VLA defines a delayed semantic-control interface that conditions action generation on delayed vision-language semantic states and explicit latency metadata.<n>To support realistic evaluation, we present DynaNav, a physics-accurate, photo-realistic simulation suite for language-guided navigation in dynamic environments.
  arXiv  Detail & Related papers  (2026-02-02T18:47:49Z)
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future [125.92052530850425]
  Vision-Language-Action (VLA) frameworks integrate perception with language-grounded decision making.<n>VLA frameworks offer a pathway toward more interpretable, generalizable, and human-aligned driving policies.<n>This work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-18T16:57:44Z)
• Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving [46.99350914451702]
  Reasoning-VLA achieves state-of-the-art performance, superior generalization capability, and the excellent inference speed reported to date.<n>We consolidate eight publicly available autonomous driving datasets into a standardized, Chain-of-language reasoning-based, and easy-to-use data format for model training.
  arXiv  Detail & Related papers  (2025-11-25T04:40:11Z)
• A Survey on Vision-Language-Action Models for Autonomous Driving [26.407082158880204]
  Vision-Language-Action (VLA) paradigms integrate visual perception, natural language understanding, and control within a single policy.<n>Researchers in autonomous driving are actively adapting these methods to the vehicle domain.<n>This survey offers the first comprehensive overview of VLA for Autonomous Driving.
  arXiv  Detail & Related papers  (2025-06-30T16:50:02Z)
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning [37.176428069948535]
  Vision-Language-Action (VLA) models have shown promise for end-to-end autonomous driving.<n>Current VLA models struggle with physically infeasible action outputs, complex model structures, or unnecessarily long reasoning.<n>We propose AutoVLA, a novel VLA model that unifies reasoning and action generation within a single autoregressive generation model.
  arXiv  Detail & Related papers  (2025-06-16T17:58:50Z)
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models [89.44024245194315]
  We introduce a method that incorporates explicit visual chain-of-thought (CoT) reasoning into vision-language-action models (VLAs)<n>We introduce CoT-VLA, a state-of-the-art 7B VLA that can understand and generate visual and action tokens.<n>Our experimental results demonstrate that CoT-VLA achieves strong performance, outperforming the state-of-the-art VLA model by 17% in real-world manipulation tasks and 6% in simulation benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T22:23:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.