A Vision-Language-Action Model with Visual Prompt for OFF-Road Autonomous Driving

• URL: http://arxiv.org/abs/2601.03519v2
• Date: Mon, 12 Jan 2026 02:37:04 GMT
• Title: A Vision-Language-Action Model with Visual Prompt for OFF-Road Autonomous Driving
• Abstract summary: Efficient trajectory planning in off-road terrains presents a formidable challenge for autonomous vehicles.<n>This paper proposes OFF-EMMA, a novel end-to-end multimodal framework to overcome the deficiencies of insufficient spatial perception and unstable reasoning.<n> Experimental results on the RELLIS-3D off-road dataset demonstrate that OFF-EMMA significantly outperforms existing methods.
• Score: 14.778856339414455
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Efficient trajectory planning in off-road terrains presents a formidable challenge for autonomous vehicles, often necessitating complex multi-step pipelines. However, traditional approaches exhibit limited adaptability in dynamic environments. To address these limitations, this paper proposes OFF-EMMA, a novel end-to-end multimodal framework designed to overcome the deficiencies of insufficient spatial perception and unstable reasoning in visual-language-action (VLA) models for off-road autonomous driving scenarios. The framework explicitly annotates input images through the design of a visual prompt block and introduces a chain-of-thought with self-consistency (COT-SC) reasoning strategy to enhance the accuracy and robustness of trajectory planning. The visual prompt block utilizes semantic segmentation masks as visual prompts, enhancing the spatial understanding ability of pre-trained visual-language models for complex terrains. The COT- SC strategy effectively mitigates the error impact of outliers on planning performance through a multi-path reasoning mechanism. Experimental results on the RELLIS-3D off-road dataset demonstrate that OFF-EMMA significantly outperforms existing methods, reducing the average L2 error of the Qwen backbone model by 13.3% and decreasing the failure rate from 16.52% to 6.56%.

Related papers

• Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Driving [79.71689985927625]
  We propose IRR-Drive, an adaptive multimodal reflection framework for autonomous driving.<n>IRR-Drive explicitly models anticipated scene evolution, enabling the model to rigorously self-correct and refine its initial intent.<n>Our method achieves state-of-the-art performance on the benchmark in both PDMS and NAVSIMS.
  arXiv  Detail & Related papers  (2026-06-22T06:53:58Z)
• ParkingTransformer: LLM-Enhanced End-to-End Trajectory Planning for Autonomous Parking [13.901233386612722]
  ParkingTransformer is a novel framework that leverages multi-view perception and the scene understanding capability of Large Language Models (LLMs)<n>To compensate for the inadequate spatial reasoning ability of LLMs, we introduce 3D positional encoding to explicitly inject spatial geometric awareness.<n>Our method achieves a driving score of 61.32 in CARLA simulator and an average success rate of 88.70% in real-world experiments.
  arXiv  Detail & Related papers  (2026-06-12T05:52:01Z)
• UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving [48.461267171124945]
  We propose UniDriveVLA, a Unified Driving Vision-Language-Action model for autonomous driving.<n>It comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention.<n>It achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive.
  arXiv  Detail & Related papers  (2026-04-02T15:48:45Z)
• Multimodal Classification Network Guided Trajectory Planning for Four-Wheel Independent Steering Autonomous Parking Considering Obstacle Attributes [22.803961716308894]
  We propose a novel trajectory planning framework that employs a neural network for scene perception.<n> multimodal perception network fusing visual information and vehicle states is employed to capture semantic and contextual scene understanding.<n>A hierarchical obstacle handling strategy, which categorizes obstacles as "non-traversable", "crossable", and "drive-over", is incorporated into the node expansion process.
  arXiv  Detail & Related papers  (2025-12-21T17:45:57Z)
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail [85.47497935739936]
  Alpamayo-R1 (AR1) is a vision-language-action model that integrates Chain of Causation reasoning with trajectory planning.<n>We show AR1 achieves 12% improvement in planning accuracy on challenging cases compared to a trajectory-only baseline.<n>We plan to release AR1 models and a subset of the CoC in a future update.
  arXiv  Detail & Related papers  (2025-10-30T01:25:34Z)
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving [55.13109926181247]
  We introduce ReflectDrive, a learning-based framework that integrates a reflection mechanism for safe trajectory generation via discrete diffusion.<n>Central to our approach is a safety-aware reflection mechanism that performs iterative self-correction without gradient.<n>Our method begins with goal-conditioned trajectory generation to model multi-modal driving behaviors.
  arXiv  Detail & Related papers  (2025-09-24T13:35:15Z)
• ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving [64.12414815634847]
  Vision-Language Models (VLMs) and Driving World Models (DWMs) have independently emerged as powerful recipes addressing different aspects of this challenge.<n>We propose ImagiDrive, a novel end-to-end autonomous driving framework that integrates a VLM-based driving agent with a DWM-based scene imaginer.
  arXiv  Detail & Related papers  (2025-08-15T12:06:55Z)
• Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance [12.513296074529727]
  This paper proposes the Real-time Edge-based Autonomous Co-pilot Trajectory planner (REACT) for autonomous driving.<n>REACT is a V2X-integrated trajectory optimization framework for AD based on a fine-tuned lightweight Vision-Language Model (VLM)<n> evaluated on the DeepAccident benchmark, REACT achieves state-of-the-art performance, a 77% collision rate reduction, a 48.2% Video Panoptic Quality (VPQ), and a 0.57-second inference latency on the Jetson AGX Orin.
  arXiv  Detail & Related papers  (2025-08-01T20:16:04Z)
• VLM-RRT: Vision Language Model Guided RRT Search for Autonomous UAV Navigation [4.022717732460524]
  We propose Vision Language Model RRT (VLM-RRT), a hybrid approach that integrates the pattern recognition capabilities of Vision Language Models (VLMs) with the path-planning strengths of Rapidly-exploring Random Trees (RRT)<n>Our method biases sampling toward regions more likely to contain feasible paths, significantly improving sampling efficiency and path quality.
  arXiv  Detail & Related papers  (2025-05-29T09:15:44Z)
• JPDS-NN: Reinforcement Learning-Based Dynamic Task Allocation for Agricultural Vehicle Routing Optimization [3.559425487157277]
  The Entrance Dependent Vehicle Routing Problem (EDVRP) is a variant of the Vehicle Problem (VRP) where the scale of cities influences routing outcomes.<n>We propose a Joint Probability Distribution Sampling Neural Network (JPDS-NN) to effectively solve the EDVRP.<n>We show that JPDS-NN reduces travel distances by 48.4-65.4%, lowers fuel consumption by 14.0-17.6%, and computes two orders of magnitude faster than baseline methods.
  arXiv  Detail & Related papers  (2025-03-04T07:50:32Z)
• 3DTTNet: Multimodal Fusion-Based 3D Traversable Terrain Modeling for Off-Road Environments [10.521569910467072]
  Off-road environments are significant challenges for autonomous ground vehicles.<n>In this paper, traversable area recognition is achieved through semantic scene completion.<n>A novel multimodal method, 3DTTNet, is proposed to generate dense traversable terrain estimations.
  arXiv  Detail & Related papers  (2024-12-11T08:36:36Z)
• A Multi-Loss Strategy for Vehicle Trajectory Prediction: Combining Off-Road, Diversity, and Directional Consistency Losses [68.68514648185828]
  Trajectory prediction is essential for the safety and efficiency of planning in autonomous vehicles.<n>Current models often fail to fully capture complex traffic rules and the complete range of potential vehicle movements.<n>This study introduces three novel loss functions: Offroad Loss, Direction Consistency Error, and Diversity Loss.
  arXiv  Detail & Related papers  (2024-11-29T14:47:08Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• Monocular BEV Perception of Road Scenes via Front-to-Top View Projection [57.19891435386843]
  We present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view. Our model runs at 25 FPS on a single GPU, which is efficient and applicable for real-time panorama HD map reconstruction.
  arXiv  Detail & Related papers  (2022-11-15T13:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.