What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models

• URL: http://arxiv.org/abs/2607.00283v1
• Date: Wed, 01 Jul 2026 00:14:13 GMT
• Title: What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models
• Abstract summary: This work bridges the critical gap between perception and planning by enabling Vision-Language Models to identify and reason about hidden agents.<n>We introduce a novel framework that uses Planning KL-divergence (PKL), an information-theoretic metric, to systematically identify and rank occluded agents.<n>We apply this framework to the nuScenes dataset to create a new benchmark focused on high-impact scenarios.
• Score: 13.668330688399516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous vehicles must safely navigate complex environments where planning-critical agents may be hidden from view. Current approaches often treat all occlusions with uniform conservatism, yielding needlessly defensive driving, or they infer hidden spaces without estimating the impact on the planner. This work bridges the critical gap between perception and planning by enabling Vision-Language Models (VLMs) to identify and reason about the specific hidden agents that are most critical to the ego-vehicle's trajectory. We introduce a novel framework that uses Planning KL-divergence (PKL), an information-theoretic metric, to systematically identify and rank occluded agents based on their impact on the ego vehicle's plan. Using this planning-aware ranking, we employ an expert VLM (GPT-5) to generate rich, structured annotations that capture the visual evidence and reasoning required for this task. We apply this framework to the nuScenes dataset to create a new benchmark focused on high-impact scenarios. We conduct comprehensive experiments on a wide range of general-purpose and domain-adapted VLMs, demonstrating that fine-tuning on our PKL-guided data yields dramatic performance improvements across all models. Notably, our results show that smaller, fine-tuned models significantly outperform their much larger zero-shot counterparts, and that our PKL-guided data selection strategy improves performance by approximately 30\% over random sampling. Our work presents the first systematic approach for training VLMs to focus on planning-critical occlusions, enabling more semantically grounded and efficient risk assessment in autonomous driving.

Related papers

• DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving [17.749010844640388]
  Reward models play a pivotal role in reinforcement learning (RL) and multi-modal trajectory selection for autonomous driving.<n>While Vision-Language Models (VLMs) have demonstrated feasibility as reward models in other domains, their effectiveness in driving tasks remains underexplored.<n>We bridge this gap by introducing DriveReward, a reasoning trajectory evaluation dataset rigorously labeled via temporally-grounded visual guidance.
  arXiv  Detail & Related papers  (2026-06-07T09:05:49Z)
• Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance [108.46436073194546]
  We argue that effective planning should be local, visually grounded, internally generated, and directly aligned with action.<n>We propose Afford-VLA, a unified framework that internalizes task-conditioned affordance as an explicit visual planning interface within VLA models.
  arXiv  Detail & Related papers  (2026-05-22T20:43:47Z)
• Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Models [1.462945824459286]
  Vision-language models (VLMs) integrate visual recognition and language reasoning, functioning without task-specific training data.<n>Our results pave the way for transparent and effective ODD-based perception in safety-critical applications.
  arXiv  Detail & Related papers  (2026-05-08T12:17:56Z)
• CogRail: Benchmarking VLMs in Cognitive Intrusion Perception for Intelligent Railway Transportation Systems [29.385460126069386]
  We introduce a novel benchmark, CogRail, which integrates curated datasets with cognitively driven question-answer annotations.<n>Building upon this benchmark, we conduct a systematic evaluation of state-of-the-art visual-language models.<n>We propose a joint fine-tuning framework that integrates three core tasks, position perception, movement prediction, and threat analysis.
  arXiv  Detail & Related papers  (2026-01-14T16:36:26Z)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving [52.02379432801349]
  We propose SGDrive, a novel framework that structures the VLM's representation learning around driving-specific knowledge hierarchies.<n>Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition.
  arXiv  Detail & Related papers  (2026-01-09T08:55:42Z)
• RealDrive: Retrieval-Augmented Driving with Diffusion Models [42.6467760755688]
  Learning-based planners generate human-like driving behaviors by learning to reason about nuanced interactions from data.<n>Data-driven approaches often struggle with rare, safety-critical scenarios and offer limited controllability over the generated trajectories.<n>We propose RealDrive, a Retrieval-Augmented Generation framework that initializes a diffusion-based planning policy by retrieving the most relevant expert demonstrations from the training dataset.
  arXiv  Detail & Related papers  (2025-05-30T17:15:03Z)
• Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling [74.41886258801209]
  We propose a two-stage trajectory planning framework that decouples principle alignment from behavior learning.<n>Plan-R1 significantly improves planning safety and feasibility, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-05-23T09:22:19Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems.<n>We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics.<n>We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision [17.36342349850825]
  Vision-language models (VLMs) as teachers to enhance training by providing additional supervision.<n>VLM-AD achieves significant improvements in planning accuracy and reduced collision rates on the nuScenes dataset.
  arXiv  Detail & Related papers  (2024-12-19T01:53:36Z)
• The efficacy of Neural Planning Metrics: A meta-analysis of PKL on nuScenes [77.83263286776938]
  A high-performing object detection system plays a crucial role in autonomous driving (AD) The performance, typically evaluated in terms of mean Average Precision, does not take into account orientation and distance of the actors in the scene. Recently, Philion et al. proposed a neural planning metric (PKL), based on the KL divergence of a planner's trajectory and the groundtruth route.
  arXiv  Detail & Related papers  (2020-10-19T09:32:48Z)
• The Importance of Prior Knowledge in Precise Multimodal Prediction [71.74884391209955]
  Roads have well defined geometries, topologies, and traffic rules. In this paper we propose to incorporate structured priors as a loss function. We demonstrate the effectiveness of our approach on real-world self-driving datasets.
  arXiv  Detail & Related papers  (2020-06-04T03:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.