Emergent Coordination and Phase Structure in Independent Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2511.23315v1
• Date: Fri, 28 Nov 2025 16:14:31 GMT
• Title: Emergent Coordination and Phase Structure in Independent Multi-Agent Reinforcement Learning
• Abstract summary: We run large-scale experiments across environment size L and agent density.<n>We construct a phase map using two axes - the cooperative success rate (CSR) and a stability index derived from TD-error variance.<n>We show that decentralized MARL exhibits a coherent phase structure governed by the interaction between scale, density, and kernel drift.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A clearer understanding of when coordination emerges, fluctuates, or collapses in decentralized multi-agent reinforcement learning (MARL) is increasingly sought in order to characterize the dynamics of multi-agent learning systems. We revisit fully independent Q-learning (IQL) as a minimal decentralized testbed and run large-scale experiments across environment size L and agent density rho. We construct a phase map using two axes - the cooperative success rate (CSR) and a stability index derived from TD-error variance - revealing three distinct regimes: a coordinated and stable phase, a fragile transition region, and a jammed or disordered phase. A sharp double Instability Ridge separates these regimes and corresponds to persistent kernel drift, the time-varying shift of each agent's effective transition kernel induced by others' policy updates. Synchronization analysis further shows that temporal alignment is required for sustained cooperation, and that competition between drift and synchronization generates the fragile regime. Removing agent identifiers eliminates drift entirely and collapses the three-phase structure, demonstrating that small inter-agent asymmetries are a necessary driver of drift. Overall, the results show that decentralized MARL exhibits a coherent phase structure governed by the interaction between scale, density, and kernel drift, suggesting that emergent coordination behaves as a distribution-interaction-driven phase phenomenon.

Related papers

• From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts [61.695980784491475]
  Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds.<n>We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity.
  arXiv  Detail & Related papers  (2026-09-06T05:43:22Z)
• Effective Covariance Dynamics in Solvable High-Dimensional GANs [2.639628765667673]
  We study a solvable high-dimensional model of generative adversarial network (GAN) training in which a linear generator learns a low-dimensional subspace from data.<n>We prove that the microscopic training process converges, in the high-dimensional limit, to deterministic ordinary differential equations governed by this effective covariance.
  arXiv  Detail & Related papers  (2026-06-25T16:33:54Z)
• High-Dimensional Latents Should Be Diagnosed Through Phase Structure [56.362776482614976]
  We study autoencoder and variational-autoencoder latent spaces through the lens of spin-glass theory.<n>We formalize a latent-space spin-glass dictionary for a fixed decoder.<n>We show that driving the latent system toward the edge-of-stability of the topological trivialization regime has concrete downstream consequences.
  arXiv  Detail & Related papers  (2026-05-23T06:17:23Z)
• Continuous-Time Distribution Matching for Few-Step Diffusion Distillation [57.28746398500951]
  We introduce Continuous-Time Distribution Matching (CDM), migrating the DMD framework from discrete anchoring to continuous optimization.<n>CDM achieves this through two continuous-time designs. First, we replace the fixed discrete schedule with a dynamic continuous schedule of random length.<n>Second, we propose a continuous-time alignment objective that performs active off-trajectory matching on latents extrapolated via the student's velocity field.
  arXiv  Detail & Related papers  (2026-05-07T14:56:39Z)
• Structural Equivalence and Learning Dynamics in Delayed MARL [2.2302915692528367]
  We establish the equivalence between Observation Delay (OD) and Action Delay (AD) in cooperative partially observable multi-agent systems.<n>We show that both systems generate identical admissible joint-policy sets and their induced state-action-observation trajectories are identical in distribution.<n>This formally generalizes existing infinite-horizon single-agent results to any-horizon partially observable cooperative multi-agent problems.
  arXiv  Detail & Related papers  (2026-05-05T23:07:09Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving [37.09409794849228]
  Topology-conditioned Stackelberg Coordination is a learning framework for decentralized interactive driving under communication-free execution.<n>We show that TSC achieves superior performance over representative MARL baselines across key metrics.
  arXiv  Detail & Related papers  (2026-02-27T10:46:41Z)
• CTHA: Constrained Temporal Hierarchical Architecture for Stable Multi-Agent LLM Systems [0.0]
  Multi-time-scale agent architectures have extended the ubiquitous single-loop paradigm by introducing temporal hierarchies with distinct cognitive layers.<n>We propose Constrained Temporal Hierarchical Architecture (CTHA) to restore coordination stability, while incorporating principled arbitration mechanisms to ensure coherent decision-making.<n>CTHA is effective for complex task execution at scale, offering 47% reduction in failure cascades, 2.3x improvement in sample efficiency, and superior scalability compared to unconstrained hierarchical baselines.
  arXiv  Detail & Related papers  (2026-01-09T08:03:14Z)
• Agent Drift: Quantifying Behavioral Degradation in Multi-Agent LLM Systems Over Extended Interactions [0.0]
  Agent drift is the progressive degradation of agent behavior, decision quality, and inter-agent coherence over extended interaction sequences.<n>We introduce the Agent Stability Index (ASI), a novel composite metric for quantifying drift across twelve dimensions.<n>We show how unchecked agent drift can lead to substantial reductions in task completion accuracy and increased human intervention requirements.
  arXiv  Detail & Related papers  (2026-01-07T18:37:26Z)
• Investigation of quantum chaos in local and non-local Ising models [0.0]
  We investigate quantum chaos within Ising spin chains subjected to transverse and longitudinal fields.<n>We show that systems with non-local interactions display a stronger propensity toward chaos, even when the non-local couplings are weak.<n>Our findings underscore the role of non-local interactions in accelerating the onset of chaos and modifying dynamical complexity in quantum spin chains.
  arXiv  Detail & Related papers  (2025-12-25T15:25:01Z)
• Morphing Through Time: Diffusion-Based Bridging of Temporal Gaps for Robust Alignment in Change Detection [51.56484100374058]
  We introduce a modular pipeline that improves spatial and temporal robustness without altering existing change detection networks.<n>A diffusion module synthesizes intermediate morphing frames that bridge large appearance gaps, enabling RoMa to estimate stepwise correspondences.<n>Experiments on LEVIR-CD, WHU-CD, and DSIFN-CD show consistent gains in both registration accuracy and downstream change detection.
  arXiv  Detail & Related papers  (2025-11-11T08:40:28Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails [103.05296856071931]
  We identify the Alignment Tipping Process (ATP), a critical post-deployment risk unique to self-evolving Large Language Model (LLM) agents.<n>ATP arises when continual interaction drives agents to abandon alignment constraints established during training in favor of reinforced, self-interested strategies.<n>Our experiments show that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states.
  arXiv  Detail & Related papers  (2025-10-06T14:48:39Z)
• Multimodal Quantitative Measures for Multiparty Behaviour Evaluation [6.709251546882382]
  We introduce a unified, intervention-driven framework for objective assessment of multiparty social behaviour in skeletal motion data.<n>We validate metric sensitivity through three theory-driven perturbations.<n>Mixed-effects analyses reveal predictable, joint-independent shifts.
  arXiv  Detail & Related papers  (2025-08-01T13:46:12Z)
• Genuine multipartite entanglement as a probe of many-body localization in disordered spin chains with Dzyaloshinskii-Moriya interactions [0.0]
  We show that the quenched average genuine multipartite entanglement (GME) can approach its maximum value in the ergodic phase of a disordered quantum spin model.<n>In contrast, GME vanishes in the many-body localized (MBL) phase, both in equilibrium and in the long-time dynamical steady state.
  arXiv  Detail & Related papers  (2025-07-30T15:57:37Z)
• Locality Preserving Markovian Transition for Instance Retrieval [59.16243976912006]
  We introduce the Locality Preserving Markovian Transition (LPMT) framework, which employs a long-term thermodynamic transition process with multiple states for accurate manifold distance measurement.<n>The proposed LPMT first integrates diffusion processes across separate graphs using Bidirectional Collaborative Diffusion (BCD) to establish strong similarity relationships.<n>After, Locality State Embedding (LSE) encodes each instance into a distribution for enhanced local consistency.<n>These distributions are interconnected via the Thermodynamic Markovian Transition (TMT) process, enabling efficient global retrieval while maintaining local effectiveness.
  arXiv  Detail & Related papers  (2025-06-05T16:07:31Z)
• Complexity transitions in chaotic quantum systems: Nonstabilizerness, entanglement, and fractal dimension in SYK and random matrix models [16.646029651172658]
  We study the transition from chaotic to integrable phases in several paradigmatic models.<n>We compare three complementary complexity markers -- fractal dimension, von Neumann entanglement entropy, and stabilizer R'enyi entropy.
  arXiv  Detail & Related papers  (2025-05-14T18:07:18Z)
• Offline Multi-agent Reinforcement Learning via Score Decomposition [51.23590397383217]
  offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts.<n>This work is the first work to explicitly address the distributional gap between offline and online MARL.
  arXiv  Detail & Related papers  (2025-05-09T11:42:31Z)
• Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis [55.13545823385091]
  Federated reinforcement learning (FedRL) enables collaborative learning while preserving data privacy by preventing direct data exchange between agents.<n>In real-world applications, each agent may experience slightly different transition dynamics, leading to inherent model mismatches.<n>We show that even moderate levels of information sharing significantly mitigate environment-specific errors.
  arXiv  Detail & Related papers  (2025-03-21T18:06:28Z)
• State Dependent Spread Complexity Dynamics in Many-Body Localization Transition [0.0]
  We characterize the Many-Body Localization (MBL) phase transition using the dynamics of spread complexity and inverse participation ratio in the Krylov space. Our work sheds light on the efficacy of Krylov space dynamics in understanding phase transitions in quantum many-body systems.
  arXiv  Detail & Related papers  (2024-09-03T18:00:11Z)
• Localization, fractality, and ergodicity in a monitored qubit [0.5892638927736115]
  We study the statistical properties of a single two-level system (qubit) subject to repetitive ancilla-based measurements. This setup is a fundamental minimal model for exploring the interplay between the unitary dynamics of the system and the nonunitaryity introduced by quantum measurements.
  arXiv  Detail & Related papers  (2023-10-03T12:10:30Z)
• Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling [67.75095378830694]
  We develop a two-stage odometry estimation network, where we obtain the ego-motion by estimating a set of sub-region transformations. In this paper, we aim to alleviate the influence of unreliable structures in training, inference and mapping phases. Our two-frame odometry outperforms the previous state of the arts by 16%/12% in terms of translational/rotational errors.
  arXiv  Detail & Related papers  (2022-02-27T12:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.