RIFT: Closed-Loop RL Fine-Tuning for Realistic and Controllable Traffic Simulation

• URL: http://arxiv.org/abs/2505.03344v1
• Date: Tue, 06 May 2025 09:12:37 GMT
• Title: RIFT: Closed-Loop RL Fine-Tuning for Realistic and Controllable Traffic Simulation
• Authors: Keyu Chen, Wenchao Sun, Hao Cheng, Sifa Zheng,
• Abstract summary: We introduce a dual-stage AV-centered simulation framework that conducts open-loop imitation learning pre-training in a data-driven simulator to capture trajectory-level realism and multimodality.<n>In the fine-tuning stage, we propose RIFT, a simple yet effective closed-loop RL fine-tuning strategy that preserves the trajectory-level multimodality.<n>Extensive experiments demonstrate that RIFT significantly improves the realism and controllability of generated traffic scenarios.
• Score: 8.952198850855426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving both realism and controllability in interactive closed-loop traffic simulation remains a key challenge in autonomous driving. Data-driven simulation methods reproduce realistic trajectories but suffer from covariate shift in closed-loop deployment, compounded by simplified dynamics models that further reduce reliability. Conversely, physics-based simulation methods enhance reliable and controllable closed-loop interactions but often lack expert demonstrations, compromising realism. To address these challenges, we introduce a dual-stage AV-centered simulation framework that conducts open-loop imitation learning pre-training in a data-driven simulator to capture trajectory-level realism and multimodality, followed by closed-loop reinforcement learning fine-tuning in a physics-based simulator to enhance controllability and mitigate covariate shift. In the fine-tuning stage, we propose RIFT, a simple yet effective closed-loop RL fine-tuning strategy that preserves the trajectory-level multimodality through a GRPO-style group-relative advantage formulation, while enhancing controllability and training stability by replacing KL regularization with the dual-clip mechanism. Extensive experiments demonstrate that RIFT significantly improves the realism and controllability of generated traffic scenarios, providing a robust platform for evaluating autonomous vehicle performance in diverse and interactive scenarios.

Related papers

• ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation [72.78362530982109]
  ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, is a framework that decouples exploration from commitment.<n>We show that naive LLM-based simulators struggle to capture rare but high-impact failure modes.<n>We introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions.
  arXiv  Detail & Related papers  (2026-02-02T06:33:22Z)
• Dynamics-Decoupled Trajectory Alignment for Sim-to-Real Transfer in Reinforcement Learning for Autonomous Driving [37.61391230495979]
  Reinforcement learning (RL) has shown promise in robotics, but deploying RL on real vehicles remains challenging.<n> Factors such as tire characteristics, road surface conditions, aerodynamic disturbances, and vehicle load make it infeasible to model real-world dynamics accurately.<n>We present a framework that decouples motion planning from vehicle control through a spatial and temporal alignment strategy.
  arXiv  Detail & Related papers  (2025-11-10T14:45:24Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving [55.13109926181247]
  We introduce ReflectDrive, a learning-based framework that integrates a reflection mechanism for safe trajectory generation via discrete diffusion.<n>Central to our approach is a safety-aware reflection mechanism that performs iterative self-correction without gradient.<n>Our method begins with goal-conditioned trajectory generation to model multi-modal driving behaviors.
  arXiv  Detail & Related papers  (2025-09-24T13:35:15Z)
• MoRe-ERL: Learning Motion Residuals using Episodic Reinforcement Learning [31.10756488629562]
  MoRe-ERL is a framework that combines Episodic Reinforcement Learning (ERL) and residual learning.<n>MoRe-ERL identifies trajectory segments requiring modification while preserving critical task-related maneuvers.<n>It generates smooth residual adjustments using B-Spline-based movement primitives.
  arXiv  Detail & Related papers  (2025-08-02T15:28:11Z)
• Robustness of Reinforcement Learning-Based Traffic Signal Control under Incidents: A Comparative Study [4.731967623788092]
  Reinforcement learning-based traffic signal control (RL-TSC) has emerged as a promising approach for improving urban mobility.<n>In this study, we introduce T-REX, an open-source, SUMO-based simulation framework for training and evaluating RL-TSC methods under dynamic, incident scenarios.
  arXiv  Detail & Related papers  (2025-06-16T08:15:29Z)
• Safety-Critical Traffic Simulation with Guided Latent Diffusion Model [8.011306318131458]
  Safety-critical traffic simulation plays a crucial role in evaluating autonomous driving systems.<n>We propose a guided latent diffusion model (LDM) capable of generating physically realistic and adversarial scenarios.<n>Our work provides an effective tool for realistic safety-critical scenario simulation, paving the way for more robust evaluation of autonomous driving systems.
  arXiv  Detail & Related papers  (2025-05-01T13:33:34Z)
• TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning [61.33599727106222]
  TeLL-Drive is a hybrid framework that integrates a Teacher LLM to guide an attention-based Student DRL policy.<n>A self-attention mechanism then fuses these strategies with the DRL agent's exploration, accelerating policy convergence and boosting robustness.
  arXiv  Detail & Related papers  (2025-02-03T14:22:03Z)
• Causal Composition Diffusion Model for Closed-loop Traffic Generation [31.52951126032351]
  We introduce the Causal Compositional Diffusion Model (CCDiff), a structure-guided diffusion framework to address these challenges.<n>We first formulate the learning of controllable and realistic closed-loop simulation as a constrained optimization problem.<n>Then, CCDiff maximizes controllability while adhering to realism by automatically identifying and injecting causal structures directly into the diffusion process.
  arXiv  Detail & Related papers  (2024-12-23T19:20:29Z)
• SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout [30.098214039454927]
  Realistic and interactive scene simulation is a key prerequisite for autonomous vehicle (AV) development.<n>We present SceneDiffuser, a scene-level diffusion prior designed for traffic simulation.<n>Novel diffusion denoising paradigm amortizes the computational cost of denoising over future simulation steps.
  arXiv  Detail & Related papers  (2024-12-05T18:06:53Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [57.278726604424556]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers.<n>Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy.<n>We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation [53.85002640149283]
  Key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge. This study identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models.
  arXiv  Detail & Related papers  (2023-09-01T19:29:53Z)
• Model-Based Reinforcement Learning with Isolated Imaginations [61.67183143982074]
  We propose Iso-Dream++, a model-based reinforcement learning approach. We perform policy optimization based on the decoupled latent imaginations. This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild.
  arXiv  Detail & Related papers  (2023-03-27T02:55:56Z)
• Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses [130.15554653948897]
  In vehicular mixed reality (MR) Metaverse, distance between physical and virtual entities can be overcome. Large-scale traffic and driving simulation via realistic data collection and fusion from the physical world is difficult and costly. We propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations.
  arXiv  Detail & Related papers  (2023-02-16T16:54:10Z)
• Guided Conditional Diffusion for Controllable Traffic Simulation [42.198185904248994]
  Controllable and realistic traffic simulation is critical for developing and verifying autonomous vehicles. Data-driven approaches generate realistic and human-like behaviors, improving transfer from simulated to real-world traffic. We develop a conditional diffusion model for controllable traffic generation (CTG) that allows users to control desired properties of trajectories at test time.
  arXiv  Detail & Related papers  (2022-10-31T14:44:59Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.