Action Emergence from Streaming Intent

• URL: http://arxiv.org/abs/2605.12622v2
• Date: Thu, 14 May 2026 17:59:01 GMT
• Title: Action Emergence from Streaming Intent
• Abstract summary: We formalize action emergence as a target capability for end-to-end autonomous driving.<n>We show that previous paradigms cannot deliver action emergence.<n>We propose Streaming Intent as a mechanism that makes driving intent semantically streamed.
• Score: 32.90086331157582
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We formalize action emergence as a target capability for end-to-end autonomous driving: the ability to generate physically feasible, semantically appropriate, and safety-compliant actions in arbitrary, long-tail traffic scenes through scene-conditioned reasoning rather than retrieval or interpolation of learned scene-action mappings. We show that previous paradigms cannot deliver action emergence: autoregressive trajectory decoders collapse the inherently multimodal future into a single averaged output, while diffusion and flow-matching generators express multimodality but are not steerable by reasoned intent. We propose Streaming Intent as a concrete way to approach action emergence: a mechanism that makes driving intent (i) semantically streamed through a continuous chain-of-thought that causally derives the intent from scene understanding, and (ii) temporally streamed across clips so that intent commitments remain coherent along the driving horizon. We realize Streaming Intent in a VLA model we call SI (Streaming Intent). SI autoregressively decodes a four-step chain-of-thought and emits an intent token; the decoded intent then drives classifier-free guidance (CFG) on a flow-matching action head, requiring only two denoising steps to generate the final trajectory. On the Waymo End-to-End benchmark, SI achieves competitive aggregate performance, with an RFS score of 7.96 on the validation set and 7.74 on the test set. Beyond aggregate metrics, the model demonstrates -- to our knowledge for the first time in a fully end-to-end VLA -- intent-faithful controllability: for a fixed scene, varying the intent class at inference yields qualitatively distinct yet consistently high-quality plans, arising purely from data-driven learning without any pre-built trajectory bank or hand-coded post-hoc selector.

Related papers

• AnchorVLA: Bridging Discrete Decisions and Continuous Trajectories for Vision-Language-Action Planning [13.949868529278504]
  AnchorVLA is a hierarchical decision-anchored VLA planning framework.<n>It uses trajectory-pattern anchors as an explicit interface between high-level VLA reasoning and continuous trajectory execution.<n>It achieves a state-of-the-art Success Rate of 77.28 and a competitive Driving Score of 89.92.
  arXiv  Detail & Related papers  (2026-07-03T10:38:30Z)
• Multimodal Action Diffusion for Robust End-to-End Autonomous Driving [0.0]
  Action Diffusion Transformer (ADT) is an anchor-free diffusion transformer trained with a MSE objective.<n>Rather than committing to a single deterministic command, ADT generates K action candidates and selects the most suitable one at inference.<n>ADT surpasses previous state-of-the-art on the challenging closed-loop Bench2Drive benchmark while achieving ten times lower latency.
  arXiv  Detail & Related papers  (2026-06-01T11:35:35Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving [54.57163800903507]
  We present MindVLA-U1, the first unified streaming Vision-Language-Action architecture for autonomous driving.<n>A unified VLM backbone produces autoregressive language tokens and flow-matching continuous action trajectories in a single forward pass over one shared representation.<n>On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time.
  arXiv  Detail & Related papers  (2026-05-12T18:09:42Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving [10.83468363371212]
  End-to-end autonomous driving requires models to understand traffic scenes, infer driving intent, and generate executable motion plans.<n>We propose VECTOR-DRIVE, a tightly coupled vision-language-action framework built on Qwen2.5-VL-3B.<n>On Bench2Drive, VECTOR-DRIVE achieves 88.91 Driving Score and outperforms representative end-to-end and VLA-based baselines.
  arXiv  Detail & Related papers  (2026-05-09T09:34:50Z)
• AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving [36.82081211127408]
  OURS is an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model.<n>We show that OURS achieves competitive performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-03-16T05:50:31Z)
• LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers [15.4994260281059]
  We introduce LAD-Drive, a generative framework that disentangles high-level intention from low-level spatial planning.<n>LAD-Drive employs an action decoder to infer a probabilistic meta-action distribution, establishing an explicit belief state that preserves the nuanced intent typically lost by one-hot encodings.<n>Extensive evaluations on the LangAuto benchmark demonstrate that LAD-Drive achieves state-of-the-art results, outperforming competitive baselines by up to 59% in Driving Score.
  arXiv  Detail & Related papers  (2026-03-02T16:21:42Z)
• Unifying Language-Action Understanding and Generation for Autonomous Driving [25.23561391638388]
  Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving.<n>Existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation.<n>We introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency.
  arXiv  Detail & Related papers  (2026-03-02T04:41:10Z)
• StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding [52.55809460075286]
  We propose a StreamAgent that anticipates the temporal intervals and spatial regions expected to contain future task-relevant information.<n>We integrate question semantics and historical observations through prompting the anticipatory agent to anticipate the temporal progression of key events.<n>Our method outperforms existing methods in response accuracy and real-time efficiency, highlighting its practical value for real-world streaming scenarios.
  arXiv  Detail & Related papers  (2025-08-03T18:15:42Z)
• ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving [49.07731497951963]
  ReCogDrive is a novel Reinforced Cognitive framework for end-to-end autonomous driving.<n>We introduce a hierarchical data pipeline that mimics the sequential cognitive process of human drivers.<n>We then address the language-action mismatch by injecting the VLM's learned driving priors into a diffusion planner.
  arXiv  Detail & Related papers  (2025-06-09T03:14:04Z)
• Learning Salient Boundary Feature for Anchor-free Temporal Action Localization [81.55295042558409]
  Temporal action localization is an important yet challenging task in video understanding. We propose the first purely anchor-free temporal localization method. Our model includes (i) an end-to-end trainable basic predictor, (ii) a saliency-based refinement module, and (iii) several consistency constraints.
  arXiv  Detail & Related papers  (2021-03-24T12:28:32Z)
• IntentNet: Learning to Predict Intention from Raw Sensor Data [86.74403297781039]
  In this paper, we develop a one-stage detector and forecaster that exploits both 3D point clouds produced by a LiDAR sensor as well as dynamic maps of the environment. Our multi-task model achieves better accuracy than the respective separate modules while saving computation, which is critical to reducing reaction time in self-driving applications.
  arXiv  Detail & Related papers  (2021-01-20T00:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.