Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models

• URL: http://arxiv.org/abs/2604.08266v1
• Date: Thu, 09 Apr 2026 13:51:55 GMT
• Title: Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models
• Abstract summary: Distilling Large Language Models (LLMs) into a compact driving model offers a compelling solution.<n>We demonstrate that an efficient vision-only student model textbfOrion-Lite can even surpass the performance of its massive VLA teacher, ORION.
• Score: 16.98212985592303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Leveraging the general world knowledge of Large Language Models (LLMs) holds significant promise for improving the ability of autonomous driving systems to handle rare and complex scenarios. While integrating LLMs into Vision-Language-Action (VLA) models has yielded state-of-the-art performance, their massive parameter counts pose severe challenges for latency-sensitive and energy-efficient deployment. Distilling LLM knowledge into a compact driving model offers a compelling solution to retain these reasoning capabilities while maintaining a manageable computational footprint. Although previous works have demonstrated the efficacy of distillation, these efforts have primarily focused on relatively simple scenarios and open-loop evaluations. Therefore, in this work, we investigate LLM distillation in more complex, interactive scenarios under closed-loop evaluation. We demonstrate that through a combination of latent feature distillation and ground-truth trajectory supervision, an efficient vision-only student model \textbf{Orion-Lite} can even surpass the performance of its massive VLA teacher, ORION. Setting a new state-of-the-art on the rigorous Bench2Drive benchmark, with a Driving Score of 80.6. Ultimately, this reveals that vision-only architectures still possess significant, untapped potential for high-performance reactive planning.

Related papers

• On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning [3.2748787252933442]
  Large language models (LLMs) have recently demonstrated strong potential for autonomous vehicle motion planning.<n>We study how to effectively transfer motion planning knowledge from a large teacher LLM to a smaller, more deployable student model.
  arXiv  Detail & Related papers  (2026-04-09T08:06:19Z)
• DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA [72.9197085473598]
  DIAL is a framework bridging high-level decision making and low-level motor execution through a differentiable latent intent bottleneck.<n>A VLM-based System-2 performs latent world modeling by latent visual foresight within the VLM's native feature space.<n>A lightweight System-1 policy then decodes this predicted intent together with the current observation into precise robot actions.
  arXiv  Detail & Related papers  (2026-03-31T15:02:27Z)
• SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving [10.980525810871827]
  We propose SAMoE-VLA, a scene-adaptive Vision-Language-Action framework that conditions expert selection on structured scene representations.<n>Our key idea is to derive the MoE routing signal from bird's-eye-view (BEV) features that encapsulates traffic scene context.<n>To support temporally consistent reasoning across world-knowledge, perception, language, and action, we introduce a Conditional Cross-Modal Causal Attention mechanism.
  arXiv  Detail & Related papers  (2026-03-09T08:54:03Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning [69.36145467833498]
  We introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving.<n> evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems.
  arXiv  Detail & Related papers  (2025-12-04T05:05:41Z)
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning [37.176428069948535]
  Vision-Language-Action (VLA) models have shown promise for end-to-end autonomous driving.<n>Current VLA models struggle with physically infeasible action outputs, complex model structures, or unnecessarily long reasoning.<n>We propose AutoVLA, a novel VLA model that unifies reasoning and action generation within a single autoregressive generation model.
  arXiv  Detail & Related papers  (2025-06-16T17:58:50Z)
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving [51.47621083057114]
  SOLVE is an innovative framework that synergizes Vision-Language Models with end-to-end (E2E) models to enhance autonomous vehicle planning.<n>Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components.
  arXiv  Detail & Related papers  (2025-05-22T15:44:30Z)
• LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving [13.136505955569875]
  Vision-Language Models (VLMs) have demonstrated significant potential for end-to-end autonomous driving.<n>We introduce LightEMMA, a Lightweight End-to-End Multimodal Model for Autonomous driving.
  arXiv  Detail & Related papers  (2025-05-01T04:12:41Z)
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models [49.32669226551026]
  We propose a simple yet effective training strategy MoE-Tuning for LVLMs.<n>MoE-LLaVA, a MoE-based sparse LVLM architecture, uniquely activates only the top-k experts through routers.<n>Experiments show the significant performance of MoE-LLaVA in a variety of visual understanding and object hallucination benchmarks.
  arXiv  Detail & Related papers  (2024-01-29T08:13:40Z)
• Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models [61.28463542324576]
  Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can generate human-like outputs. We evaluate existing state-of-the-art VLMs and find that even the best-performing model is unable to demonstrate strong visual reasoning capabilities and consistency. We propose a two-stage training framework aimed at improving both the reasoning performance and consistency of VLMs.
  arXiv  Detail & Related papers  (2023-09-08T17:49:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.