CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving

• URL: http://arxiv.org/abs/2605.17284v1
• Date: Sun, 17 May 2026 06:45:53 GMT
• Title: CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving
• Abstract summary: We introduce CLAP, a location-aware adaptation framework that augments a frozen VLA driving model with per-roadblock soft prompts.<n>On the NAVSIM benchmark with various state-of-the-art VLA backbones, CLAP reduces challenging scenario planning error by 24% with no regression on normal frames.
• Score: 12.890001628695215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: End-to-end autonomous driving systems powered by Vision-Language-Action (VLA) models achieve strong performance on common driving scenarios, yet remain brittle in rare but safety-critical long-tail situations such as active construction zones and complex yielding geometries. In this paper, we present a method that addresses the long-tail challenging scenes beyond data scaling and model training. We introduce CLAP (Contrastive Latent-space Prompt optimization), a location-aware adaptation framework that augments a frozen VLA driving model with per-roadblock soft prompts, optimized from crowdsourced data and retrieved on demand via Vehicle-to-Everything (V2X) communication. Our approach rests on two observations from VLAs' latent space: (i) at the VLA's hidden-state layer, scenarios from the same roadblock cluster tightly and occupy compact regions of the latent space; and (ii) within a single roadblock, long-tail and normal frames are heavily intermixed in the latent representation, making it difficult to improve one without disturbing the other. CLAP addresses this via a two-stage pipeline: supervised contrastive learning to discover a roadblock-specific hard-scene direction, followed by directionally regularized prompt optimization that selectively improves challenging frames while preserving normal frame performance. On the NAVSIM benchmark with various state-of-the-art VLA backbones, CLAP reduces challenging scenario planning error by 24% with no regression on normal frames, significantly improving planning performance.

Related papers

• HiRAD: A Flexible Large-Scale AGV Routing System [6.059667257410882]
  We propose HiRAD, a hierarchical RL framework for continuous-space AGV routing with real-time guarantees.<n>HiRAD reduces makespan by 45 percent to 63 percent and shortens end-to-end runtime.
  arXiv  Detail & Related papers  (2026-09-09T05:50:17Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework [49.531001563853984]
  RAD-2 is a unified generator-discriminator framework for closed-loop planning.<n>It produces diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality.<n>It reduces the collision rate by 56% compared with strong diffusion-based planners.
  arXiv  Detail & Related papers  (2026-04-16T17:59:44Z)
• HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving [51.268878540511054]
  We propose HAD, an end-to-end planning framework with a Hierarchical Diffusion Policy.<n>We show that HAD achieves new state-of-the-art performance on both NAVSIM and HUGSIM.
  arXiv  Detail & Related papers  (2026-04-04T04:12:47Z)
• FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving [11.275815014211046]
  Building a learning-based closed-loop simulator for autonomous driving poses three major challenges.<n>Long-horizon temporal and cross-view consistency, mitigating autoregressive degradation under iterative self-conditioning, and satisfying low-latency inference constraints.<n>We propose FAR-Drive, a frame-level autoregressive video generation framework for autonomous driving.
  arXiv  Detail & Related papers  (2026-03-16T07:40:13Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• A Unified Candidate Set with Scene-Adaptive Refinement via Diffusion for End-to-End Autonomous Driving [11.096777574777406]
  A fixed trajectory vocabulary provides stable coverage in routine driving but often misses optimal solutions in complex interactions.<n>We propose CdDrive, which preserves the original vocabulary candidates and augments them with scene-adaptive candidates generated by vocabulary-conditioned diffusion denoising.<n>Both candidate types are jointly scored by a shared selection module, enabling reliable performance across routine and highly interactive scenarios.
  arXiv  Detail & Related papers  (2026-02-03T05:14:08Z)
• ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving [44.008287454538596]
  Vision-language models (VLMs) enrich this paradigm by introducing cross-modal priors and commonsense reasoning.<n>Current VLM-based planners face three key challenges: (i) a mismatch between discrete text reasoning and continuous control, (ii) high latency from autoregressive chain-of-thought decoding, and (iii) inefficient or non-causal planners that limit real-time deployment.<n>We propose ColaVLA, a unified vision-language-action framework that transfers reasoning from text to a unified latent space and couples it with a hierarchical, parallel trajectory decoder.
  arXiv  Detail & Related papers  (2025-12-28T14:06:37Z)
• Multi-Modality Driven LoRA for Adverse Condition Depth Estimation [61.525312117638116]
  We propose Multi-Modality Driven LoRA (MMD-LoRA) for Adverse Condition Depth Estimation.<n>It consists of two core components: Prompt Driven Domain Alignment (PDDA) and Visual-Text Consistent Contrastive Learning (VTCCL)<n>It achieves state-of-the-art performance on the nuScenes and Oxford RobotCar datasets.
  arXiv  Detail & Related papers  (2024-12-28T14:23:58Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes [70.08318779492944]
  We are the first to harness vanishing point (VP) priors for more effective segmentation. Our novel, efficient network for VSS, named VPSeg, incorporates two modules that utilize exactly this pair of static and dynamic VP priors.
  arXiv  Detail & Related papers  (2024-01-27T01:01:58Z)
• Integrating Higher-Order Dynamics and Roadway-Compliance into Constrained ILQR-based Trajectory Planning for Autonomous Vehicles [3.200238632208686]
  Trajectory planning aims to produce a globally optimal route for Autonomous Passenger Vehicles. Existing implementations utilizing the vehicle bicycle kinematic model may not guarantee controllable trajectories. We augment this model by higher-order terms, including the first and second-order derivatives of curvature and longitudinal jerk.
  arXiv  Detail & Related papers  (2023-09-25T22:30:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.