One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

• URL: http://arxiv.org/abs/2608.12253v2
• Date: Mon, 17 Aug 2026 17:48:56 GMT
• Title: One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
• Abstract summary: Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior.<n>We show that this approach systematically fails to generalize, and trace the failure to simulator collapse.<n>We propose two complementary solutions, one at inference time and one at training time.
• Score: 75.34038228759182
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $τ^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

Related papers

• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL [55.05743310621117]
  Co-RL is a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers.<n>We show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops.
  arXiv  Detail & Related papers  (2026-08-18T01:16:02Z)
• Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning [65.62918039166772]
  We propose a zero-sum minimax game between a model player and an adversarial policy player.<n>We show that our approach reduces prediction error in strategically important regions by $1.5$-$2.2times$ and enables policies trained purely in simulation to match near-optimal real-world performance.
  arXiv  Detail & Related papers  (2026-05-27T19:31:37Z)
• Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents [28.85410475505536]
  Persona Policies (PPol) is a control layer that induces realistic behavioral variation in user simulators.<n>PPol produces a diverse population of human-like personas for any task in the domain.<n>Agents trained with PPol are more robust to challenging, out-of-distribution behaviors, improving task success by +17% relative to training only on existing simulated interactions.
  arXiv  Detail & Related papers  (2026-05-13T02:16:51Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• PolySim: Bridging the Sim-to-Real Gap for Humanoid Control via Multi-Simulator Dynamics Randomization [53.7088694598817]
  We introduce PolySim, a WBC training platform that integrates multiple heterogeneous simulators.<n>Theoretically, we show that PolySim yields a tighter upper bound on simulator inductive bias than single-simulator training.
  arXiv  Detail & Related papers  (2025-10-02T06:31:42Z)
• Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games [42.74003740156243]
  We address the problem of agile 1v1 quadrotor pursuit-evasion, where a pursuer and an evader learn to outmaneuver each other.<n>We propose an Asynchronous Multi-Stage Population-Based (AMSPB) algorithm to tackle these issues.<n>Within this framework, we train neural network controllers that output either velocity commands or body rates with collective thrust.
  arXiv  Detail & Related papers  (2025-06-03T13:19:23Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [57.278726604424556]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers.<n>Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy.<n>We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems [43.31078296862647]
  Long-term user engagement (LTE) optimization in sequential recommender systems (SRS) is suited by reinforcement learning (RL) RL has its shortcomings, particularly requiring a large number of online samples for exploration. We present a simulator-based recommender policy training approach, Simulation-to-Recommendation (Sim2Rec)
  arXiv  Detail & Related papers  (2023-05-03T19:21:25Z)
• Zero-shot Sim2Real Adaptation Across Environments [45.44896435487879]
  We propose a Reverse Action Transformation (RAT) policy which learns to imitate simulated policies in the real-world. RAT can then be deployed on top of a Universal Policy Network to achieve zero-shot adaptation to new environments.
  arXiv  Detail & Related papers  (2023-02-08T11:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.