PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning

• URL: http://arxiv.org/abs/2608.01201v1
• Date: Sun, 02 Aug 2026 12:33:52 GMT
• Title: PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning
• Abstract summary: End-to-end autonomous driving systems integrate perception, prediction, and planning into a single differentiable architecture.<n>Recent works have integrated vision-language model (VLM) supervision for latent features to address this, yielding substantial empirical gains.<n>Our investigation into this methodology reveals that the performance gains stem not from VLM reasoning capabilities, as previously assumed, but rather from the latent connections forged between the E2E AD model and ground-truth (GT) data during training.
• Score: 18.741741219314836
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end autonomous driving (E2E AD) systems integrate perception, prediction, and planning into a single differentiable architecture. While these models show great promise, their standard training often relies on output-only supervision, which can lead to weak gradients for the hidden layers of increasingly complex models. Recent works have integrated vision-language model (VLM) supervision for latent features to address this, yielding substantial empirical gains, yet leaving the underlying theoretical mechanisms poorly understood. Our investigation into this methodology reveals that the resulting performance gains stem not from VLM reasoning capabilities, as previously assumed, but rather from the latent connections forged between the E2E AD model and ground-truth (GT) data during training. Building on this insight, we propose a probabilistic deep supervision framework that regularizes intermediate latent representations directly from GT data. By treating model latents as reparameterizable distributions, we optimize the architecture via the Evidence Lower Bound (ELBO). Our evaluations conducted on the nuScenes dataset demonstrate that supervising trajectory-related latents with future GT paths consistently improves planning performance. Using identical training data and E2E architectures, our method achieves an 8% reduction in planning L2 error and a 3% decrease in collision rates compared to competitive vectorized baselines, all while incurring negligible computational overhead.

Related papers

• AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution [61.593935260052795]
  Post-training viaSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs)<n>Existing paradigms often reach a performance bottleneck due to the limitations of static data.<n>We propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution.
  arXiv  Detail & Related papers  (2026-05-25T08:26:34Z)
• GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control [0.0]
  We introduce GIRL (Generative Imagination Reinforcement Learning), a latent world-model framework that addresses this failure mode with two key components.<n> GIRL reduces latent rollout drift by 38 to 61 percent across tasks relative to DreamerV3, improves return, and requires fewer environment interactions on long-horizon tasks.<n>A distilled-prior variant reduces overhead and improves computational efficiency relative to the full model.
  arXiv  Detail & Related papers  (2026-04-08T17:14:21Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Simplicity Prevails: The Emergence of Generalizable AIGI Detection in Visual Foundation Models [15.709482146201283]
  A simple linear classifier trained on the frozen features of modern Vision Foundation Models establishes a new state-of-the-art.<n>We show that this baseline matches specialized detectors on standard benchmarks but also decisively outperforms them on in-the-wild datasets.<n>We conclude by advocating for a paradigm shift in AI forensics, moving from overfitting on static benchmarks to harnessing the evolving world knowledge of foundation models for real-world reliability.
  arXiv  Detail & Related papers  (2026-02-02T07:20:02Z)
• PILOT: Planning via Internalized Latent Optimization Trajectories for Large Language Models [51.43746425777865]
  Large Language Models (LLMs) often lack the capacity to formulate global strategies, leading to error propagation in long-horizon tasks.<n>We propose PILOT, a framework designed to internalize the strategic oversight of large models into intrinsic Latent Guidance.
  arXiv  Detail & Related papers  (2026-01-07T12:38:56Z)
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning [69.36145467833498]
  We introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving.<n> evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems.
  arXiv  Detail & Related papers  (2025-12-04T05:05:41Z)
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL [29.682761652941963]
  Probe, Learn, Distill (PLD) is a three-stage plug-and-play framework that improves vision-language-action models.<n>PLD achieves near-saturated 99% task success on LIBERO, over 50% gains in SimplerEnv, and 100% success on real-world Franka and YAM arm manipulation tasks.
  arXiv  Detail & Related papers  (2025-10-30T06:24:04Z)
• DIFFUMA: High-Fidelity Spatio-Temporal Video Prediction via Dual-Path Mamba and Diffusion Enhancement [5.333662480077316]
  We release the Chip Dicing Lane dataset (CHDL), the first public temporal image dataset dedicated to the semiconductor wafer dicing process.<n>We propose DIFFUMA, an innovative dual-path prediction architecture specifically designed for such fine-grained dynamics.<n>Experiments demonstrate that DIFFUMA significantly outperforms existing methods, reducing the Mean Squared Error (MSE) by 39% and improving the Similarity (SSIM) from 0.926 to a near-perfect 0.988.
  arXiv  Detail & Related papers  (2025-07-09T10:51:54Z)
• Towards foundational LiDAR world models with efficient latent flow matching [3.971158433168816]
  A single pre-trained model can achieve up to 11% absolute improvement over training from scratch and outperforms training from scratch in 30/36 of our comparisons.<n>We propose a latent conditional flow matching framework that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods.
  arXiv  Detail & Related papers  (2025-06-30T00:16:55Z)
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving [51.47621083057114]
  SOLVE is an innovative framework that synergizes Vision-Language Models with end-to-end (E2E) models to enhance autonomous vehicle planning.<n>Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components.
  arXiv  Detail & Related papers  (2025-05-22T15:44:30Z)
• Latent Diffusion Planning for Imitation Learning [78.56207566743154]
  Latent Diffusion Planning (LDP) is a modular approach consisting of a planner and inverse dynamics model.<n>By separating planning from action prediction, LDP can benefit from the denser supervision signals of suboptimal and action-free data.<n>On simulated visual robotic manipulation tasks, LDP outperforms state-of-the-art imitation learning approaches.
  arXiv  Detail & Related papers  (2025-04-23T17:53:34Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.