Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving

• URL: http://arxiv.org/abs/2606.20274v1
• Date: Thu, 18 Jun 2026 14:18:01 GMT
• Title: Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving
• Abstract summary: Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories.<n>We propose Lagrange, an open-vocabulary, computationally sparse driving framework based on Masked Latent Fields (MLF)<n>We introduce an intent-driven masked cross-attention module that temporally filters irrelevant entities, decoding the attended tokens into an implicit continuous energy field defined over spatial coordinates.
• Score: 12.828441606176447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories. Existing paradigms face a distinct dichotomy between representational efficiency and generalization capacity. Dense models (e.g., occupancy networks), while geometrically robust, incur critical computational bottlenecks and struggle with high-level semantic reasoning. Conversely, sparse, query-based planners are efficient but reliant on closed-set definitions, rendering them vulnerable to out-of-distribution (OOD) events. Although recent Vision-Language-Action (VLA) models offer open-vocabulary reasoning, their autoregressive, discrete token generation fundamentally conflicts with the continuous, high-frequency control requirements of vehicle dynamics. To address this, we propose Lagrange, an open-vocabulary, computationally sparse driving framework based on Masked Latent Fields (MLF). Rather than relying on dense volumetric reconstructions or closed-set query mechanisms, Lagrange exploits Vision-Language Models (VLMs) to encode class-agnostic object proposals into continuous semantic visual tokens. We introduce an intent-driven masked cross-attention module that temporally filters irrelevant entities, decoding the attended tokens into an implicit continuous energy field defined over spatial coordinates. By framing decision-making as a Lagrangian action minimization problem spanning this energy field, we enforce strict compliance with vehicle kinematics while executing collision avoidance. Extensive offline evaluations on both standard (nuScenes) and long-tail (CODA) benchmarks demonstrate that Lagrange establishes a promising framework for robust, interpretable, and kinematically feasible open-world autonomy.

Related papers

• S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving [8.813976817688605]
  Vision-Language Models (VLMs) have demonstrated remarkable potential for high-level reasoning in autonomous driving.<n>VLMs struggle to generate precise, low-level control actions due to mismatch between discrete language tokens and continuous trajectory planning.<n>We propose the S-squared-VLA, which explicitly decouples the semantic and spatial streams in Vision-Language-Action models.
  arXiv  Detail & Related papers  (2026-07-15T15:07:02Z)
• World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning [60.88859946006306]
  Adversa World Modeling is a theoretically grounded multi-agent self-play fine-tuning framework.<n>In inner minimization, the planner's predictive world model is converted into a role-conditioned adversary that learns sparse, scene-adaptive attack coalitions.<n> Experiments on the nuPlan and InterPlan benchmarks demonstrate that our method generates transferable adversarial interactions.
  arXiv  Detail & Related papers  (2026-07-12T07:44:16Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving [8.228888952282917]
  We propose textbfVLA-DriverRAG, a framework that grounds in explicit, structure-aware historical knowledge.<n>We abstract sensory inputs into semantic graphs via a textitVisual-to-Scenario mechanism, effectively filtering visual noise.<n>Experiments on the Bench2Drive benchmark establish a new state-of-the-art, achieving a Driving Score of 89.12.
  arXiv  Detail & Related papers  (2026-05-01T05:50:00Z)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation [61.18260993245354]
  Chain-of-Thought (CoT) reasoning is a powerful driver of trajectory prediction in VLA-based autonomous driving.<n>We present OneVL, a unified VLA and World Model framework that routes reasoning through compact latent tokens supervised by dual auxiliary decoders.<n>OneVL becomes the first latent CoT method to surpass explicit CoT, delivering state-of-the-art accuracy at answer-only latency.
  arXiv  Detail & Related papers  (2026-04-20T16:37:22Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges [2.76240219662896]
  This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving.<n>The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics.
  arXiv  Detail & Related papers  (2026-03-10T01:56:17Z)
• Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusion [23.834662472392694]
  Masked Vision-Language-Action Diffusion for Autonomous Driving (MVLAD-AD) is a novel framework designed to bridge the gap between efficient planning and semantic explainability.<n>We introduce a discrete action tokenization strategy that constructs a compact codebook of kinematically feasible waypoints from real-world driving distributions.<n>Experiments on nuScenes and derived benchmarks demonstrate that MVLAD-AD achieves superior efficiency and outperforms state-of-the-art autoregressive and diffusion baselines in planning precision.
  arXiv  Detail & Related papers  (2026-02-24T05:59:10Z)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving [52.02379432801349]
  We propose SGDrive, a novel framework that structures the VLM's representation learning around driving-specific knowledge hierarchies.<n>Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition.
  arXiv  Detail & Related papers  (2026-01-09T08:55:42Z)
• Multi-Robot Motion Planning from Vision and Language using Heat-Inspired Diffusion [6.286759951136632]
  Language-Conditioned Heat-Inspired Diffusion (LCHD) is an end-to-end vision-based framework that generates language-conditioned, collision-free trajectories.<n> LCHD integrates CLIP-based semantic priors with a collision-avoiding diffusion kernel serving as a physical inductive bias.<n> LCHD consistently outperforms prior diffusion-based planners in success rate, while reducing planning latency.
  arXiv  Detail & Related papers  (2025-12-15T08:43:13Z)
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving [55.13109926181247]
  We introduce ReflectDrive, a learning-based framework that integrates a reflection mechanism for safe trajectory generation via discrete diffusion.<n>Central to our approach is a safety-aware reflection mechanism that performs iterative self-correction without gradient.<n>Our method begins with goal-conditioned trajectory generation to model multi-modal driving behaviors.
  arXiv  Detail & Related papers  (2025-09-24T13:35:15Z)
• InVDriver: Intra-Instance Aware Vectorized Query-Based Autonomous Driving Transformer [12.441180142943328]
  InVDriver is a novel vectorized query-based system for intra-instance spatial dependencies.<n>We show that InVDriver achieves state-of-the-art performance, surpassing prior methods in both accuracy and safety.
  arXiv  Detail & Related papers  (2025-02-25T08:20:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.