Reasoning About Traversability: Language-Guided Off-Road 3D Trajectory Planning

• URL: http://arxiv.org/abs/2604.21249v1
• Date: Thu, 23 Apr 2026 03:26:42 GMT
• Title: Reasoning About Traversability: Language-Guided Off-Road 3D Trajectory Planning
• Abstract summary: Vision-Language Models (VLMs) enable high-level semantic reasoning for end-to-end autonomous driving.<n>Existing off-road datasets suffer from language annotations that are weakly aligned with vehicle actions and terrain geometry.<n>We propose a language refinement framework that restructures annotations into action-aligned pairs, enabling a VLM to generate refined scene descriptions and 3D future trajectories.
• Score: 1.6983657949644717
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While Vision-Language Models (VLMs) enable high-level semantic reasoning for end-to-end autonomous driving, particularly in unstructured environments, existing off-road datasets suffer from language annotations that are weakly aligned with vehicle actions and terrain geometry. To address this misalignment, we propose a language refinement framework that restructures annotations into action-aligned pairs, enabling a VLM to generate refined scene descriptions and 3D future trajectories directly from a single image. To further encourage terrain-aware planning, we introduce a preference optimization strategy that constructs geometry-aware hard negatives and explicitly penalizes trajectories inconsistent with local elevation profiles. Furthermore, we propose off-road-specific metrics to quantify traversability compliance and elevation consistency, addressing the limitations of conventional on-road evaluation. Experiments on the ORAD-3D benchmark demonstrate that our approach reduces average trajectory error from 1.01m to 0.97m, improves traversability compliance from 0.621 to 0.644, and decreases elevation inconsistency from 0.428 to 0.322, highlighting the efficacy of action-aligned supervision and terrain-aware optimization for robust off-road driving.

Related papers

• Local Path Planning and Obstacle Avoidance for an Omnicopter Platform [5.329751638083761]
  This paper presents a real-time local planning and obstacle avoidance module for an omnidirectional multirotor (omnicopter)<n>Our method achieves real-time feasibility through (i) local-map voxelisation, (ii) a compact sphere-based approximation of the vehicle geometry, and (iii) adaptive velocity sampling in the 6D search space.<n>We evaluate our approach in simulation across computational stress tests, dense-waypoint path tracking, and static/unknown obstacle scenarios.
  arXiv  Detail & Related papers  (2026-09-03T10:25:14Z)
• AutoPath: Learning Transferable Goal-Conditioned Stochastic Path Prior for Safe Navigation Without Human Demonstrations [103.61864552260316]
  Real-time navigation in dynamic environments requires collision-free and dynamically feasible motion under limited perception.<n>We formulate navigation as learning a transferable goal-aligned path prior that models a reusable distribution over goal-aligned geometry-consistent local paths conditioned on local observations.<n>We develop a structured prior learning framework that parameterizes local paths using a geometry-aware polar action manifold and incorporates risk-sensitive utility shaping with multi-goal distributional rollouts for stable and safety-aware planning.
  arXiv  Detail & Related papers  (2026-07-13T16:00:41Z)
• Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation [79.12557132788139]
  Vision-Language Navigation in Continuous Environments (VLN-CE) requires agents to follow natural-language instructions while navigating in real-world-like environments.<n>Most VLN-CE approach-es adopt a three-stage framework: a waypoint predictor proposes waypoints, and a navigator selects the best waypoint.<n>We introduce a novel paradigm called Trajectory Waypoint, which grounds each candidate waypoint in an executable trajectory.
  arXiv  Detail & Related papers  (2026-06-05T13:11:39Z)
• Spatially-Aware Adaptive Trajectory Optimization with Controller-Guided Feedback for Autonomous Racing [74.83272587893508]
  We present a framework for autonomous raceline optimization that combines NURBS-based trajectory representation, CMA-ES global trajectory optimization, and controller-guided spatial feedback.<n>In simulation, our approach achieves a 17.38% lap time reduction compared to a controller parametrized with maximum static acceleration.<n>On real hardware, tested with different tire compounds ranging from high to low friction, we obtain a 7.60% lap time improvement without explicitly parametrizing friction.
  arXiv  Detail & Related papers  (2026-02-17T15:10:44Z)
• A Vision-Language-Action Model with Visual Prompt for OFF-Road Autonomous Driving [14.778856339414455]
  Efficient trajectory planning in off-road terrains presents a formidable challenge for autonomous vehicles.<n>This paper proposes OFF-EMMA, a novel end-to-end multimodal framework to overcome the deficiencies of insufficient spatial perception and unstable reasoning.<n> Experimental results on the RELLIS-3D off-road dataset demonstrate that OFF-EMMA significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2026-01-07T02:08:18Z)
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail [85.47497935739936]
  Alpamayo-R1 (AR1) is a vision-language-action model that integrates Chain of Causation reasoning with trajectory planning.<n>We show AR1 achieves 12% improvement in planning accuracy on challenging cases compared to a trajectory-only baseline.<n>We plan to release AR1 models and a subset of the CoC in a future update.
  arXiv  Detail & Related papers  (2025-10-30T01:25:34Z)
• Boundary-Guided Trajectory Prediction for Road Aware and Physically Feasible Autonomous Driving [11.436186697804835]
  This paper proposes a novel framework that formulates trajectory prediction as a constrained regression guided by permissible driving directions and their boundaries.<n>We evaluate our approach on the Argoverse-2 dataset against the HPTR baseline.
  arXiv  Detail & Related papers  (2025-05-10T19:21:00Z)
• 3DTTNet: Multimodal Fusion-Based 3D Traversable Terrain Modeling for Off-Road Environments [10.521569910467072]
  Off-road environments are significant challenges for autonomous ground vehicles.<n>In this paper, traversable area recognition is achieved through semantic scene completion.<n>A novel multimodal method, 3DTTNet, is proposed to generate dense traversable terrain estimations.
  arXiv  Detail & Related papers  (2024-12-11T08:36:36Z)
• A Multi-Loss Strategy for Vehicle Trajectory Prediction: Combining Off-Road, Diversity, and Directional Consistency Losses [68.68514648185828]
  Trajectory prediction is essential for the safety and efficiency of planning in autonomous vehicles.<n>Current models often fail to fully capture complex traffic rules and the complete range of potential vehicle movements.<n>This study introduces three novel loss functions: Offroad Loss, Direction Consistency Error, and Diversity Loss.
  arXiv  Detail & Related papers  (2024-11-29T14:47:08Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation [64.84996994779443]
  We propose a novel Affordances-Oriented Planner for continuous vision-language navigation (VLN) task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-07-08T12:52:46Z)
• DATT: Deep Adaptive Trajectory Tracking for Quadrotor Control [62.24301794794304]
  Deep Adaptive Trajectory Tracking (DATT) is a learning-based approach that can precisely track arbitrary, potentially infeasible trajectories in the presence of large disturbances in the real world. DATT significantly outperforms competitive adaptive nonlinear and model predictive controllers for both feasible smooth and infeasible trajectories in unsteady wind fields. It can efficiently run online with an inference time less than 3.2 ms, less than 1/4 of the adaptive nonlinear model predictive control baseline.
  arXiv  Detail & Related papers  (2023-10-13T12:22:31Z)
• Integrating Higher-Order Dynamics and Roadway-Compliance into Constrained ILQR-based Trajectory Planning for Autonomous Vehicles [3.200238632208686]
  Trajectory planning aims to produce a globally optimal route for Autonomous Passenger Vehicles. Existing implementations utilizing the vehicle bicycle kinematic model may not guarantee controllable trajectories. We augment this model by higher-order terms, including the first and second-order derivatives of curvature and longitudinal jerk.
  arXiv  Detail & Related papers  (2023-09-25T22:30:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.