Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulation

• URL: http://arxiv.org/abs/2605.09153v1
• Date: Sat, 09 May 2026 20:31:34 GMT
• Title: Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulation
• Abstract summary: We propose a hierarchical architecture that goes beyond self-play by combining high-level multi-agent interaction reasoning with low-level continuous trajectory realization.<n>Experiments on a SUMO-based urban network demonstrate that the proposed framework achieves smoothness superior control and safety compared to self-play and passive imitation baselines.
• Score: 8.157621070273736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Closed-loop traffic simulation requires agents that are both scalable and behaviorally realistic. Recent self-play reinforcement learning approaches demonstrate strong scalability, but their equilibrium strategies fail to capture the socially aware behaviors of real human drivers. We propose a hierarchical architecture that goes beyond self-play by combining high-level multi-agent interaction reasoning with low-level continuous trajectory realization. Specifically, a Stackelberg-style Multi-Agent Reinforcement Learning (MARL) module generates interaction-aware intention commands. These commands condition a low-level continuous motion module, translating the strategic intent into physically consistent, scene-responsive control sequences. To mitigate distribution shift in closed-loop deployment, we introduce a hybrid co-training scheme combining MARL with auxiliary recovery supervision. Experiments on a SUMO-based urban network demonstrate that the proposed framework achieves superior control smoothness and safety compared to self-play and passive imitation baselines, while maintaining competitive traffic efficiency.

Related papers

• World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning [60.88859946006306]
  Adversa World Modeling is a theoretically grounded multi-agent self-play fine-tuning framework.<n>In inner minimization, the planner's predictive world model is converted into a role-conditioned adversary that learns sparse, scene-adaptive attack coalitions.<n> Experiments on the nuPlan and InterPlan benchmarks demonstrate that our method generates transferable adversarial interactions.
  arXiv  Detail & Related papers  (2026-07-12T07:44:16Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• ForceFlow: Learning to Feel and Act via Contact-Driven Flow Matching [53.30290192030814]
  ForceFlow is a force-aware reactive framework built upon flow matching.<n>We show that ForceFlow achieves a 37% success rate improvement over the strong baseline ForceVLA.<n>It also exhibits accurate force signal prediction and superior performance in contact force self-regulation.
  arXiv  Detail & Related papers  (2026-05-11T13:27:00Z)
• Learning Reactive Dexterous Grasping via Hierarchical Task-Space RL Planning and Joint-Space QP Control [50.28263951510334]
  We propose a hybrid hierarchical control framework for reactive dexterous grasping.<n>The proposed framework explicitly decouples high-level spatial intent from low-level joint execution.<n>We extensively validate the proposed framework through a rigorous simulation-to-reality pipeline.
  arXiv  Detail & Related papers  (2026-05-05T04:49:38Z)
• Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-System [15.427164345561417]
  Vision-Language-Action (VLA) models are a promising paradigm for generalist robotic manipulation.<n>We introduce Libra-VLA, a novel Coarse-to-Fine Dual-System VLA architecture.<n>We show that our approach offers a scalable, robust, and responsive solution for open-world manipulation.
  arXiv  Detail & Related papers  (2026-04-27T19:02:46Z)
• FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving [11.275815014211046]
  Building a learning-based closed-loop simulator for autonomous driving poses three major challenges.<n>Long-horizon temporal and cross-view consistency, mitigating autoregressive degradation under iterative self-conditioning, and satisfying low-latency inference constraints.<n>We propose FAR-Drive, a frame-level autoregressive video generation framework for autonomous driving.
  arXiv  Detail & Related papers  (2026-03-16T07:40:13Z)
• TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving [37.09409794849228]
  Topology-conditioned Stackelberg Coordination is a learning framework for decentralized interactive driving under communication-free execution.<n>We show that TSC achieves superior performance over representative MARL baselines across key metrics.
  arXiv  Detail & Related papers  (2026-02-27T10:46:41Z)
• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning [59.63038625580992]
  Existing imitation learning approaches often fail to model realistic traffic behaviors.<n>We propose DecompGAIL, which explicitly decomposes realism into ego-map and ego-neighbor components.<n>DecompGAIL achieves state-of-the-art performance on the WOMD Sim Agents 2025 benchmark.
  arXiv  Detail & Related papers  (2025-10-08T11:46:39Z)
• RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation [13.319344167881383]
  We introduce a dual-stage AV-centric simulation framework that conducts imitation learning pre-training in a data-driven simulator.<n>We then learn fine-tuning in a physics-based simulator to enhance style-level controllability.<n>In the fine-tuning stage, we propose RIFT, a novel group-relative RL fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-05-06T09:12:37Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.