Fail2Drive: Benchmarking Closed-Loop Driving Generalization

• URL: http://arxiv.org/abs/2604.08535v1
• Date: Thu, 09 Apr 2026 17:59:18 GMT
• Title: Fail2Drive: Benchmarking Closed-Loop Driving Generalization
• Abstract summary: Fail2Drive is the first paired-route benchmark for generalization in CARLA.<n>Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into diagnostics.<n>Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space.
• Score: 24.431701691830046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8\%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .

Related papers

• Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling [65.03665015319457]
  We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two modes: action-class miscalibration and action-execution failure.<n>We validate it on a panel of tool-calling models across multiple multi-turn benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T09:08:15Z)
• SDVDiag: Multimodal Causal Discovery for Online Diagnosis in Software-defined Vehicles [0.35232085374661287]
  SDVDiag is a multimodal causal-discovery pipeline that fuses log-based and metric-based service representations into a shared embedding space.<n>This paper presents SDVDiag, a multimodal causal-discovery pipeline that fuses log-based and metric-based service representations into a shared embedding space.
  arXiv  Detail & Related papers  (2026-06-14T02:54:45Z)
• Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models [50.22099309218635]
  In autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation.<n>We present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving.
  arXiv  Detail & Related papers  (2026-04-01T11:38:46Z)
• Misbehavior Forecasting for Focused Autonomous Driving Systems Testing [2.7733556309376692]
  Existing bug-finding techniques are either unreliable or expensive.<n>We propose Foresee, a technique that identifies near misses using a misbehavior forecaster.<n>Foresee performs local fuzzing in the neighborhood of each candidate near miss to surface previously unknown failures.
  arXiv  Detail & Related papers  (2025-12-21T17:17:49Z)
• Failure Prediction at Runtime for Generative Robot Policies [6.375597233389154]
  Early failure prediction during runtime is essential for deploying robots in human-centered and safety-critical environments.<n>We propose FIPER, a framework for failure prediction for generative robot policies that does not require failure data.<n>Our results demonstrate that FIPER better distinguishes actual failures from benign OOD situations and predicts failures more accurately and earlier than existing methods.
  arXiv  Detail & Related papers  (2025-10-10T15:09:27Z)
• CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine [73.74077186298523]
  CoReVLA is a continual learning framework for autonomous driving.<n>It improves the performance in long-tail scenarios through a dual-stage process of data Collection and behavior Refinement.<n>CoReVLA achieves a Driving Score (DS) of 72.18 and a Success Rate (SR) of 50%, outperforming state-of-the-art methods by 7.96 DS and 15% SR under long-tail, safety-critical scenarios.
  arXiv  Detail & Related papers  (2025-09-19T13:25:56Z)
• SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment [49.86376148975563]
  Large language models (LLMs) have revolutionized natural language processing through their capabilities in understanding and executing diverse tasks.<n> supervised fine-tuning, particularly in Retrieval-Augmented Generation (RAG) scenarios, often leads to catastrophic forgetting.<n>We propose SelfAug, a self-distribution alignment method that aligns input sequence logits to preserve the model's semantic distribution.
  arXiv  Detail & Related papers  (2025-09-04T06:50:47Z)
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving [59.705635382104454]
  We present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner.<n>We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.
  arXiv  Detail & Related papers  (2024-06-06T09:12:30Z)
• Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap [109.46323960299146]
  We study vision architectures with respect to the open-loop to closed-loop causality gap. We show that under proper training guidelines, all vision models perform indistinguishably well on in-distribution deployment. Our results imply that the causality gap can be solved in situation one with our proposed training guideline with any modern network architecture.
  arXiv  Detail & Related papers  (2022-10-09T16:56:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.