Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

• URL: http://arxiv.org/abs/2605.00066v1
• Date: Thu, 30 Apr 2026 09:27:57 GMT
• Title: Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive
• Abstract summary: Open-loop evaluation offers fast, reproducible assessment of autonomous driving planners.<n>Traditional open-loop metrics exhibit no reliable correlation with closed-loop Driving Score.
• Score: 19.12252168142987
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-loop evaluation offers fast, reproducible assessment of autonomous driving planners, but its ability to predict real closed-loop driving performance remains questionable. Prior work has shown that traditional open-loop metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE) exhibit no reliable correlation with closed-loop Driving Score. In this paper, we ask whether the more recent, safety-aware open-loop metrics introduced by NAVSIM~v2 can bridge this gap. By systematically cross-referencing published results from 15 state-of-the-art methods across NAVSIM (open-loop) and Bench2Drive (closed-loop), we compile a paired dataset of open-loop sub-metrics and closed-loop performance, yielding 8 methods with complete paired data. Our analysis reveals three key findings: (1) the aggregate NAVSIM PDM Score shows a strong positive but non-monotonic correlation with Bench2Drive Driving Score, with clear ranking inversions; (2) among individual NAVSIM sub-metrics, Ego Progress (EP) is the strongest single predictor of closed-loop success, substantially exceeding the safety-critical collision metric NC; (3) the safety-progress trade-off manifests differently in open-loop and closed-loop: methods that maximize safety at the expense of progress rank highly in NAVSIM but underperform in closed-loop due to timeout and slow-driving penalties. We further demonstrate that a much simpler 3-metric formula matches the predictive power of the full 5-metric PDMS at the same Spearman $ρ{=}0.90$ on our paired sample of $n{=}8$ methods, suggesting that within current state-of-the-art methods -- where TTC and Comfort approach saturation -- these two sub-metrics add little marginal information for closed-loop ranking. Additionally, we identify the snowball effect -- where small open-loop deviations compound into closed-loop failures -- as a candidate mechanism for the residual gap.

Related papers

• CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving [66.77752555295355]
  We present CLEAR, a system that enables closed-loop training using Reinforcement Learning (RL) at scale for E2E-AD.<n>We show that with a simple reward, CLEAR significantly outperforms previous methods and sets new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-07-03T00:34:50Z)
• Riemannian MeanFlow for One-Step Generation on Manifolds [54.09734511705173]
  Flow Matching enables simulation-free training of generative models.<n>MeanFlow can be extended to manifold-valued generation where velocities lie in location-dependent tangent spaces.<n>Experiments on spheres, tori, and SO(3) demonstrate competitive one-step sampling with improved quality-efficiency trade-offs and substantially reduced sampling cost.
  arXiv  Detail & Related papers  (2026-03-11T12:41:46Z)
• Residual RL--MPC for Robust Microrobotic Cell Pushing Under Time-Varying Flow [0.6053202932413693]
  Contact-rich micromanipulation in microfluidic flow is challenging because small disturbances can break pushing contact and induce large lateral drift.<n>We study planar cell pushing with a magnetic rolling microrobot that tracks a waypoint-sampled reference curve under time-varying Poiseuille flow.<n>We propose a hybrid controller that augments a nominal MPC with a learned residual policy trained by SAC.
  arXiv  Detail & Related papers  (2026-03-05T18:20:07Z)
• Accelerating Single-Pass SGD for Generalized Linear Prediction [10.482631815569919]
  We study generalized linear prediction under a streaming setting, where each iteration uses only one fresh data point for a gradient-level update.<n>We propose the first algorithm that successfully incorporates momentum via a novel data-dependent proximal method, achieving dual-momentum acceleration.
  arXiv  Detail & Related papers  (2026-03-02T15:04:00Z)
• Pseudo-Simulation for Autonomous Driving [66.1981253104508]
  Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations.<n>Real-world evaluation is often challenging due to safety concerns and a lack of realism.<n>Open-loop evaluation relies on metrics that generally overlook compounding errors.
  arXiv  Detail & Related papers  (2025-06-04T17:57:53Z)
• Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training [64.16445087751039]
  Hydra-NeXt is a novel multi-branch planning framework that unifies trajectory prediction, control prediction, and a trajectory refinement network in one model.<n> Hydra-NeXt surpasses the previous state-of-the-art by 22.98 DS and 17.49 SR, marking a significant advancement in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-15T07:42:27Z)
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving [59.705635382104454]
  We present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner.<n>We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.
  arXiv  Detail & Related papers  (2024-06-06T09:12:30Z)
• Bi-directional Loop Closure for Visual SLAM [0.0]
  We propose an approach for bi-directional loop closure. This will provide us with the capability to relocalize to a location even when traveling in the opposite direction. We present a technique to select training data from large datasets in order to make them usable for the bi-directional problem.
  arXiv  Detail & Related papers  (2022-04-01T14:06:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.