Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models

• URL: http://arxiv.org/abs/2602.04184v1
• Date: Wed, 04 Feb 2026 03:44:56 GMT
• Title: Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
• Abstract summary: doScenes is the first real-world dataset linking free-form instructions to nuScenes ground-truth motion.<n>We adapt OpenEMMA, an open-source MLLM-based end-to-end driving framework.<n>We integrate doScenes directives as passenger-style prompts within OpenEMMA's vision-language interface.
• Score: 2.2727733134290813
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Instruction-grounded driving, where passenger language guides trajectory planning, requires vehicles to understand intent before motion. However, most prior instruction-following planners rely on simulation or fixed command vocabularies, limiting real-world generalization. doScenes, the first real-world dataset linking free-form instructions (with referentiality) to nuScenes ground-truth motion, enables instruction-conditioned planning. In this work, we adapt OpenEMMA, an open-source MLLM-based end-to-end driving framework that ingests front-camera views and ego-state and outputs 10-step speed-curvature trajectories, to this setting, presenting a reproducible instruction-conditioned baseline on doScenes and investigate the effects of human instruction prompts on predicted driving behavior. We integrate doScenes directives as passenger-style prompts within OpenEMMA's vision-language interface, enabling linguistic conditioning before trajectory generation. Evaluated on 849 annotated scenes using ADE, we observe that instruction conditioning substantially improves robustness by preventing extreme baseline failures, yielding a 98.7% reduction in mean ADE. When such outliers are removed, instructions still influence trajectory alignment, with well-phrased prompts improving ADE by up to 5.1%. We use this analysis to discuss what makes a "good" instruction for the OpenEMMA framework. We release the evaluation prompts and scripts to establish a reproducible baseline for instruction-aware planning. GitHub: https://github.com/Mi3-Lab/doScenes-VLM-Planning

Related papers

• InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation [24.530150557492934]
  We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation.<n>InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks.<n>Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts.
  arXiv  Detail & Related papers  (2026-08-24T08:53:41Z)
• SG-WAM: Text-Grounded and Spatial-aware Semantic Guidance for World-Action Models [16.792979070880147]
  We propose SG-WAM, a semantic guidance method for world-action models.<n>We use a vision-language model (VLM) as a semantic planner to enhance the instruction-grounding capacity of world-action models.
  arXiv  Detail & Related papers  (2026-08-09T17:59:35Z)
• AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation [78.80536515102305]
  Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment.<n>We propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism.<n>Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training.
  arXiv  Detail & Related papers  (2026-05-21T17:58:26Z)
• Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles [40.195332742819865]
  This study proposes an instruction-realization framework that leverages a large language model (LLM) to interpret instructions.<n>It generates executable scripts that schedule multiple model predictive control (MPC)-based motion planners based on real-time feedback, and converts planned trajectories into control signals.
  arXiv  Detail & Related papers  (2026-04-09T09:32:21Z)
• ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving [1.6961250145588112]
  We introduce ICR-Drive, a diagnostic framework for instruction counterfactual robustness in language-conditioned autonomous driving.<n>We replay identical CARLA routes under matched simulator configurations and seeds to isolate performance changes attributable to instruction language.<n>Experiments on LMDrive and BEVDriver show that minor instruction changes can induce substantial performance drops and distinct failure modes.
  arXiv  Detail & Related papers  (2026-04-07T03:24:22Z)
• Vega: Learning to Drive with Natural Language Instructions [93.49357278917223]
  We propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning.<n>We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions.<n>Our method achieves superior planning performance and exhibits strong instruction-following abilities.
  arXiv  Detail & Related papers  (2026-03-26T17:59:56Z)
• DRIV-EX: Counterfactual Explanations for Driving LLMs [5.315914253152163]
  Large language models (LLMs) are increasingly used as reasoning engines in autonomous driving.<n>We propose to study their decision process through counterfactual explanations.<n>We introduce DRIV-EX, a method that leverages gradient-based optimization on continuous embeddings.
  arXiv  Detail & Related papers  (2026-02-28T15:12:51Z)
• LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries [30.732526921367835]
  LangForce is a novel framework that enforces instruction following via Bayesian decomposition.<n>We show that LangForce significantly improves generalization without requiring new data.
  arXiv  Detail & Related papers  (2026-01-21T17:15:22Z)
• Align2Act: Instruction-Tuned Models for Human-Aligned Autonomous Driving [0.0]
  We propose Align2Act, a motion planning framework that transforms instruction-tuned language models into interpretable planners aligned with human behavior.<n>By fine-tuning LLaMA-2-7B with LoRA on one million scenarios from the nuPlan dataset, our method achieves an open-loop score of 85.17 and closed-loop scores of 70.31 and 66.96 on Test14-random.<n>Unlike prior work focused on synthetic or open-loop settings, we demonstrate improved planning quality and human-likeness on the real-world nuPlan closed-loop benchmark.
  arXiv  Detail & Related papers  (2025-10-12T08:50:34Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction [95.6266030753644]
  Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.<n>Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies.<n>We propose OTTER, a novel VLA architecture that leverages existing alignments through explicit, text-aware visual feature extraction.
  arXiv  Detail & Related papers  (2025-03-05T18:44:48Z)
• doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation [0.0]
  doScenes is a novel dataset designed to facilitate research on human-vehicle instruction interactions.<n>DoScenes bridges the gap between instruction and driving response, enabling context-aware and adaptive planning.
  arXiv  Detail & Related papers  (2024-12-08T11:16:47Z)
• Instruct Large Language Models to Drive like Humans [33.219883052634614]
  We propose an InstructDriver method to transform large language models into motion planners. We derive driving instruction data based on human logic. We then employ an interpretable InstructChain module to further reason the final planning.
  arXiv  Detail & Related papers  (2024-06-11T14:24:45Z)
• From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning [63.63840740526497]
  We investigate how instruction tuning adjusts pre-trained models with a focus on intrinsic changes. The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models. Our findings reveal three significant impacts of instruction tuning.
  arXiv  Detail & Related papers  (2023-09-30T21:16:05Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)
• A Picture is Worth a Thousand Words: Language Models Plan from Pixels [53.85753597586226]
  Planning is an important capability of artificial agents that perform long-horizon tasks in real-world environments. In this work, we explore the use of pre-trained language models (PLMs) to reason about plan sequences from text instructions in embodied visual environments.
  arXiv  Detail & Related papers  (2023-03-16T02:02:18Z)
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [111.33545170562337]
  We investigate the possibility of grounding high-level tasks, expressed in natural language, to a chosen set of actionable steps. We find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions.
  arXiv  Detail & Related papers  (2022-01-18T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.