When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit

• URL: http://arxiv.org/abs/2608.04896v1
• Date: Wed, 05 Aug 2026 14:22:23 GMT
• Title: When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit
• Abstract summary: Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when a logged human reference fails a compliance channel.<n>We audit this risk in NAVSIM v2.2 original scene single-stage scoring.
• Score: 2.823425836233287
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not. Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when the logged human reference fails a compliance channel. When agent and reference share an unstable rollout transformation, this rule can propagate shared reference failures into broad compliance credit. We audit this risk in NAVSIM v2.2 original scene single-stage scoring. Under the affected documented-stack condition on the audited numerical backend, the route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed over the complete 12,146-token navtest split. A fresh installation following the public specification reproduces rollout divergence on a fixed 32-token diagnostic set. A same-source dependency stack control and an exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit. On a 450-token control pool, replacing only the solver eliminates rollout divergence and restores blind-last ordering while keeping forgiveness enabled. Thus, the numerical instability is the direct trigger. Reference-conditioned forgiveness propagates the resulting shared reference failures into compliance credit. We contribute an audit protocol requiring score basis and stack disclosure, blind probes, overwrite reporting, and rollout stability tests before using such scores for defensive driving claims.

Related papers

• FunnelAudit: Responsibility Auditing in Multi-Route Recommender Systems [6.810073531792462]
  Multi-route recommender systems combine retrieval, allocation, fusion, and ranking, making individual inclusions and exclusions difficult to audit.<n>We introduce FunnelAudit, an executable framework for incident-level responsibility auditing.<n>We instantiate the framework in two-stage, nine-route funnels using fixed union, weighted quota allocation, or weighted reciprocal-rank fusion, followed by SASRec ranking.
  arXiv  Detail & Related papers  (2026-09-07T03:01:07Z)
• When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide [0.0]
  We benchmark six estimators across five datasets and two known-effect sweeps, and validate the mechanisms against a non-simulated paired reference.<n>Sharpening a logger from the target's own score barely moves overlap over the tested range; action-level disagreement collapses it.<n>Cross-fitting the outcome nuisance alone leaves the reuse bias in place and makes it worse.
  arXiv  Detail & Related papers  (2026-08-12T18:10:10Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents [5.91817208489443]
  We study the verifier-acceptance-deployment gap.<n>This gap refers to the discrepancy between an agent's self-authored verification signal and a sealed deployment evaluation.<n>We introduce a Sealed Exogenous Acceptance Loop (SEAL)
  arXiv  Detail & Related papers  (2026-07-27T11:45:14Z)
• Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs [4.721990925113432]
  Aligned language models routinely misreport under non-evidential pressure.<n>We study the two demands, resist (ignore forbidden pressure) and update (follow licensed evidence)<n>We causally localize low-rank report coordinates for answer, confidence, and caveat.
  arXiv  Detail & Related papers  (2026-07-14T17:28:25Z)
• Verification of Adaptive Agentic Controllers through Finite Rule Revision [0.0]
  This paper formulates a bounded verification protocol for adaptive agentic controllers represented by finite symbolic rules.<n> diagnostic failures are mapped to predefined rule-level edits, including rule addition, rule deletion, and priority revision.<n> repaired controllers are then evaluated on held-out simulation seeds or cloned initial states.
  arXiv  Detail & Related papers  (2026-07-07T13:38:01Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• Benchmarking Recursive-Collapse Warning Claims Under Matched False-Positive Control [0.0]
  Recursive systems can enter collapse-like regimes before overt failure becomes visible.<n>We introduce Loopzero, a claim-bounded benchmark framework for testing whether failures follow a directional telemetry pattern.<n>We evaluate the bridge on two frozen public-artifact benchmarks.
  arXiv  Detail & Related papers  (2026-05-29T20:12:42Z)
• Auditing Privacy in Multi-Tenant RAG under Account Collusion [1.253312107729806]
  Multi-tenant retrieval-augmented generation services advertise per-account differential privacy as the operative leakage boundary.<n>We identify same-index multi-account collusion as a privacy-boundary failure.<n>We design the first audit protocol that operates against unmodified RAG deployments.
  arXiv  Detail & Related papers  (2026-05-19T13:41:59Z)
• Counterfactual Likelihood Tests for Indirect Influence in Private Reasoning Channels [51.56484100374058]
  This paper presents a counterfactual likelihood test for measuring influence between private reasoning channels.<n>The method replaces an upstream private block with a length-matched donor block, holds the public token sequence and downstream target fixed, and measures the downstream target's negative-log-likelihood shift.
  arXiv  Detail & Related papers  (2026-05-18T20:27:43Z)
• IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory [73.22944697933603]
  Existing pipelines produce opaque, end-to-end outputs that expose no intermediate state for inspection.<n>We present IMPACT-Cycle, a supervisory multi-agent system that reformulates long-video understanding as a multimodal iterative claim-level maintenance.
  arXiv  Detail & Related papers  (2026-04-22T03:03:33Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.