Multi-ORFT: Stable Online Reinforcement Fine-Tuning for Multi-Agent Diffusion Planning in Cooperative Driving

• URL: http://arxiv.org/abs/2604.11734v2
• Date: Tue, 14 Apr 2026 07:22:13 GMT
• Title: Multi-ORFT: Stable Online Reinforcement Fine-Tuning for Multi-Agent Diffusion Planning in Cooperative Driving
• Abstract summary: We present Multi-ORFT, which couples scene-conditioned diffusion pre-training with stable online reinforcement post-training.<n>In pre-training, the planner uses inter-agent self-attention, cross-attention, and AdaLN-Zero-based scene conditioning.<n>In post-training, we formulate a two-level MDP that exposes step-wise reverse-relative likelihoods for online optimization.
• Score: 22.627579758896967
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Closed-loop cooperative driving requires planners that generate realistic multimodal multi-agent trajectories while improving safety and traffic efficiency. Existing diffusion planners can model multimodal behaviors from demonstrations, but they often exhibit weak scene consistency and remain poorly aligned with closed-loop objectives; meanwhile, stable online post-training in reactive multi-agent environments remains difficult. We present Multi-ORFT, which couples scene-conditioned diffusion pre-training with stable online reinforcement post-training. In pre-training, the planner uses inter-agent self-attention, cross-attention, and AdaLN-Zero-based scene conditioning to improve scene consistency and road adherence of joint trajectories. In post-training, we formulate a two-level MDP that exposes step-wise reverse-kernel likelihoods for online optimization, and combine dense trajectory-level rewards with variance-gated group-relative policy optimization (VG-GRPO) to stabilize training. On the WOMD closed-loop benchmark, Multi-ORFT reduces collision rate from 2.04% to 1.89% and off-road rate from 1.68% to 1.36%, while increasing average speed from 8.36 to 8.61 m/s relative to the pre-trained planner, and it outperforms strong open-source baselines including SMART-large, SMART-tiny-CLSFT, and VBD on the primary safety and efficiency metrics. These results show that coupling scene-consistent denoising with stable online diffusion-policy optimization improves the reliability of closed-loop cooperative driving.

Related papers

• DSIP: A Dynamic Coordination Planner for Signal-Free Intersections using Diffusion-Model-Based Multi-Agent Motion Planning [13.207796958340067]
  Traffic signal control at urban intersections introduces stop-and-go behavior, resulting in increased delays and reduced traffic efficiency.<n> trajectory-level coordination has emerged as a high-potential strategy to augment or transcend conventional phase-based management.<n>This paper proposes DSIP (Diffusion-model-based Signal-free Intersection Planner), a multi-agent motion planning framework driven by a generative diffusion process.
  arXiv  Detail & Related papers  (2026-06-29T01:35:13Z)
• Boosting Multimodal Federated Learning via Chained Modality Optimization [69.53075328069504]
  Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability.<n>To address this, we propose FedMChain, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases.<n>This phase-wise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an error-compensated regularizer.
  arXiv  Detail & Related papers  (2026-06-01T08:07:09Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework [49.531001563853984]
  RAD-2 is a unified generator-discriminator framework for closed-loop planning.<n>It produces diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality.<n>It reduces the collision rate by 56% compared with strong diffusion-based planners.
  arXiv  Detail & Related papers  (2026-04-16T17:59:44Z)
• Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning [7.804905867413516]
  Model Predictive Control (MPC) offers structured constraint handling through optimization.<n>Deep Reinforcement Learning (RL) learns adaptive behaviors from experience but often struggles with safety assurance.<n>We present an integrated MPC-RL framework to improve navigation performance in multi-agent scenarios.
  arXiv  Detail & Related papers  (2026-04-15T13:58:38Z)
• HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving [51.268878540511054]
  We propose HAD, an end-to-end planning framework with a Hierarchical Diffusion Policy.<n>We show that HAD achieves new state-of-the-art performance on both NAVSIM and HUGSIM.
  arXiv  Detail & Related papers  (2026-04-04T04:12:47Z)
• ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving [63.980630608984605]
  This paper presents ADV-0, a closed-loop min-max optimization framework that treats the interaction between driving policy (defender) and adversarial agent (attacker) as a zero-sum Markov game.<n>To make this tractable, we cast dynamic adversary evolution as iterative preference learning, efficiently approximating this optimum and offering an algorithm-agnostic solution to the game.<n> Experiments indicate that it effectively exposes diverse safety-critical failures and greatly enhances the generalizability of both learned policies and motion planners against unseen long-tail risks.
  arXiv  Detail & Related papers  (2026-03-16T12:58:31Z)
• Multi-Objective Reinforcement Learning for Large-Scale Mixed Traffic Control [3.1133295867933057]
  We propose a hierarchical framework combining multi-objective reinforcement learning for local intersection control with strategic routing for network-level coordination.<n>Our approach introduces a Conflict Threat Vector that provides agents with explicit risk signals for proactive conflict avoidance, and a queue parity penalty that ensures equitable service across all traffic streams.
  arXiv  Detail & Related papers  (2025-12-12T03:10:26Z)
• Discrete-Guided Diffusion for Scalable and Safe Multi-Robot Motion Planning [56.240199425429445]
  Multi-Robot Motion Planning (MPMP) involves generating trajectories for multiple robots operating in a shared continuous workspace.<n>While discrete multi-agent finding (MAPF) methods are broadly adopted due to their scalability, their coarse discretization trajectory quality.<n>This paper tackles limitations of two approaches by introducing discrete MAPF solvers with constrained generative diffusion models.
  arXiv  Detail & Related papers  (2025-08-27T17:59:36Z)
• Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning [10.579847782542982]
  This paper presents a decentralized policy network using multi-agent reinforcement learning for time-optimal multi-drone flight.<n>To strike a balance between flight efficiency and collision avoidance, we introduce a soft collision-free mechanism inspired by optimization-based methods.<n> Extensive simulations show that, despite slight performance trade-offs compared to single-drone systems, our multi-drone approach maintains near-time-optimal performance with a low collision rate.
  arXiv  Detail & Related papers  (2024-09-25T08:09:52Z)
• iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning [57.24340061741223]
  We introduce a distributed multi-agent reinforcement learning (MARL) algorithm that can predict trajectories and intents in dense and heterogeneous traffic scenarios. Our approach for intent-aware planning, iPLAN, allows agents to infer nearby drivers' intents solely from their local observations.
  arXiv  Detail & Related papers  (2023-06-09T20:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.