HABIT: Human Action Benchmark for Interactive Traffic in CARLA

• URL: http://arxiv.org/abs/2511.19109v1
• Date: Mon, 24 Nov 2025 13:43:39 GMT
• Title: HABIT: Human Action Benchmark for Interactive Traffic in CARLA
• Abstract summary: We introduce HABIT (Human Action Benchmark for Interactive Traffic), a high-fidelity simulation benchmark.<n>From an initial pool of approximately 30,000 retargeted motions, we curate 4,730 traffic-compatible pedestrian motions.<n>Our safety metrics, including Abbreviated Injury Scale (AIS) and False Positive Braking Rate (FPBR), reveal critical failure modes in state-of-the-art AD agents missed by prior evaluations.
• Score: 0.2905751301655124
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current autonomous driving (AD) simulations are critically limited by their inadequate representation of realistic and diverse human behavior, which is essential for ensuring safety and reliability. Existing benchmarks often simplify pedestrian interactions, failing to capture complex, dynamic intentions and varied responses critical for robust system deployment. To overcome this, we introduce HABIT (Human Action Benchmark for Interactive Traffic), a high-fidelity simulation benchmark. HABIT integrates real-world human motion, sourced from mocap and videos, into CARLA (Car Learning to Act, a full autonomous driving simulator) via a modular, extensible, and physically consistent motion retargeting pipeline. From an initial pool of approximately 30,000 retargeted motions, we curate 4,730 traffic-compatible pedestrian motions, standardized in SMPL format for physically consistent trajectories. HABIT seamlessly integrates with CARLA's Leaderboard, enabling automated scenario generation and rigorous agent evaluation. Our safety metrics, including Abbreviated Injury Scale (AIS) and False Positive Braking Rate (FPBR), reveal critical failure modes in state-of-the-art AD agents missed by prior evaluations. Evaluating three state-of-the-art autonomous driving agents, InterFuser, TransFuser, and BEVDriver, demonstrates how HABIT exposes planner weaknesses that remain hidden in scripted simulations. Despite achieving close or equal to zero collisions per kilometer on the CARLA Leaderboard, the autonomous agents perform notably worse on HABIT, with up to 7.43 collisions/km and a 12.94% AIS 3+ injury risk, and they brake unnecessarily in up to 33% of cases. All components are publicly released to support reproducible, pedestrian-aware AI research.

Related papers

• CausalDrive: Real-time Causal World Models for Autonomous Driving [60.66609721457312]
  We present CausalDrive, a controllable, real-time foundation driving world.<n>CaulDrive operates solely on the initial front-view frame, the ego-vehicle's trajectory, and a macroscopic text prompt.<n>We demonstrate its versatility across three downstream applications: (1) generative closed-loop evaluation with significantly mitigated collision artifacts, (2) large-scale Reinforcement Learning (RL) post-training driven by a Video2Reward module, and (3) real-time human-in-the-loop simulation.
  arXiv  Detail & Related papers  (2026-06-13T15:04:44Z)
• ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving [56.96697737229895]
  Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems.<n>We introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving.
  arXiv  Detail & Related papers  (2026-06-12T03:11:06Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• Autonomous Vehicle Collision Avoidance With Racing Parameterized Deep Reinforcement Learning [73.88859384645264]
  Road traffic accidents are a leading cause of fatalities worldwide.<n>In the US, human error causes 94% of crashes, resulting in over 7,000 pedestrian fatalities and $500 billion in costs annually.<n>This paper parameterizes a Deep Reinforcement Learning (DRL) collision avoidance policy out-of-Distribution.<n>Two policies are evaluated, a default uni-direction and a reversed heading variant that navigates in the opposite direction to other cars.
  arXiv  Detail & Related papers  (2026-04-17T21:11:39Z)
• Heterogeneous Self-Play for Realistic Highway Traffic Simulation [42.67401323169807]
  PHASE, Policy for Heterogeneous Agent Self-play on Expressway, is a context-aware self-play framework.<n>It supports different vehicle profiles, for example, passenger cars and articulated trailer trucks, within a single policy.<n>It achieves a 96.3% success rate and reduces ADE/FDE from 6.57/12.07 m to 2.44/5.25 m relative to a prior self-play baseline.
  arXiv  Detail & Related papers  (2026-03-31T19:39:29Z)
• Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets [7.932761533792761]
  High infraction rates remain the primary bottleneck for end-to-end (E2E) autonomous driving.<n>We develop a Video-Language-Augmented Anomaly Detector (VLAAD)<n>VLAAD is a collision-aware plug-in module that can be seamlessly integrated into existing E2E driving models.
  arXiv  Detail & Related papers  (2026-03-26T22:32:52Z)
• DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving [7.788062051923755]
  DriveVLM-RL is a neuroscience-inspired framework that integrates vision-language models into reinforcement learning.<n>The framework decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment.<n>A hierarchical reward synthesis mechanism fuses semantic signals with vehicle states, while an asynchronous training pipeline decouples expensive VLM inference from environment interaction.
  arXiv  Detail & Related papers  (2026-03-18T21:55:29Z)
• Optimization-Guided Diffusion for Interactive Scene Generation [52.23368750264419]
  We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling.<n>We show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes.<n>Our approach can also generate $5times$ more near-collision frames with a time-to-collision under three seconds.
  arXiv  Detail & Related papers  (2025-12-08T15:56:18Z)
• SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identification [1.6386429281694148]
  This paper proposes SVBRD-LLM, a framework that automatically discovers, verifies, and applies interpretable behavioral rules from real traffic videos.<n>The framework extracts vehicle trajectories using YOLOv8 and ByteTrack, computes kinematic features, and employs GPT-5 zero-shot prompting to compare autonomous and human-driven vehicles.<n> Experiments on over 1500 hours of real traffic videos show that the framework achieves 90.0% accuracy and 93.3% F1-score in autonomous vehicle identification.
  arXiv  Detail & Related papers  (2025-11-18T23:45:30Z)
• ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring [52.195295396336526]
  ZTRS (Zero-Imitation End-to-End Autonomous Driving with Trajectory Scoring) is a framework that combines the strengths of both worlds: sensor inputs without losing information and RL training for robust planning.<n>ZTRS demonstrates strong performance across three benchmarks: Navtest, Navhard, and HUGSIM.
  arXiv  Detail & Related papers  (2025-10-28T06:26:36Z)
• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning [59.63038625580992]
  Existing imitation learning approaches often fail to model realistic traffic behaviors.<n>We propose DecompGAIL, which explicitly decomposes realism into ego-map and ego-neighbor components.<n>DecompGAIL achieves state-of-the-art performance on the WOMD Sim Agents 2025 benchmark.
  arXiv  Detail & Related papers  (2025-10-08T11:46:39Z)
• MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving [85.04826012938642]
  MetAdv is a novel adversarial testing platform that enables realistic, dynamic, and interactive evaluation.<n>It supports flexible 3D vehicle modeling and seamless transitions between simulated and physical environments.<n>It enables real-time capture of physiological signals and behavioral feedback from drivers.
  arXiv  Detail & Related papers  (2025-08-04T03:07:54Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Minds on the Move: Decoding Trajectory Prediction in Autonomous Driving with Cognitive Insights [18.92479778025183]
  In driving scenarios, a vehicle's trajectory is determined by the decision-making process of human drivers.<n>Previous models fail to capture the true intentions of human drivers, leading to suboptimal performance in long-term trajectory prediction.<n>We introduce a Cognitive-Informed Transformer (CITF) that incorporates a cognitive concept, Perceived Safety, to interpret drivers' decision-making mechanisms.
  arXiv  Detail & Related papers  (2025-02-27T13:43:17Z)
• Building reliable sim driving agents by scaling self-play [3.3378669626639423]
  Simulation agents are essential for designing and testing systems that interact with humans, such as autonomous vehicles (AVs)<n>We propose scaling self-play to thousands of scenarios on the Open Motion dataset under semi-realistic limits on human perception and control.<n>We generalize to unseen test scenes, achieving a 99.8% goal completion rate with less than 0.8% combined collision and off-road incidents.
  arXiv  Detail & Related papers  (2025-02-20T16:30:45Z)
• Are you a robot? Detecting Autonomous Vehicles from Behavior Analysis [6.422370188350147]
  We present a framework that monitors active vehicles using camera images and state information in order to determine whether vehicles are autonomous. Essentially, it builds on the cooperation among vehicles, which share their data acquired on the road feeding a machine learning model to identify autonomous cars. Experiments show it is possible to discriminate the two behaviors by analyzing video clips with an accuracy of 80%, which improves up to 93% when the target state information is available.
  arXiv  Detail & Related papers  (2024-03-14T17:00:29Z)
• What Matters to Enhance Traffic Rule Compliance of Imitation Learning for End-to-End Autonomous Driving [10.191916541924813]
  We proposed P-CSG, a penalty-based imitation learning approach with contrastive-based cross semantics generation sensor fusion technologies. In this paper, we introduce three penalties - red light, stop sign, and curvature speed penalty to make the agent more sensitive to traffic rules. We conducted robustness evaluations against adversarial attacks like FGSM and Dot attacks, revealing a substantial increase in robustness compared to other baseline models.
  arXiv  Detail & Related papers  (2023-09-14T15:54:56Z)
• ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events [1.84926694477846]
  We propose a black-box testing framework that uses offline trajectories first to analyze the existing behavior of autonomous vehicles. Our experiment shows an increase in 35, 23, 48, and 50% in the occurrences of vehicle collision, road object collision, pedestrian collision, and offroad steering events.
  arXiv  Detail & Related papers  (2023-08-28T13:09:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.