Structural Equivalence and Learning Dynamics in Delayed MARL

• URL: http://arxiv.org/abs/2605.04345v1
• Date: Tue, 05 May 2026 23:07:09 GMT
• Title: Structural Equivalence and Learning Dynamics in Delayed MARL
• Abstract summary: We establish the equivalence between Observation Delay (OD) and Action Delay (AD) in cooperative partially observable multi-agent systems.<n>We show that both systems generate identical admissible joint-policy sets and their induced state-action-observation trajectories are identical in distribution.<n>This formally generalizes existing infinite-horizon single-agent results to any-horizon partially observable cooperative multi-agent problems.
• Score: 2.2302915692528367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We formally establish the equivalence between Observation Delay (OD) and Action Delay (AD) in cooperative partially observable multi-agent systems using observation-action histories. We show that both systems generate identical admissible joint-policy sets, and their induced state-action-observation trajectories are identical in distribution, leading to identical optimal solutions in Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). This formally generalizes existing infinite-horizon single-agent results to any-horizon partially observable cooperative multi-agent problems with decentralized policy execution, and allows any mixed-delay configuration to be reduced to a pure OD system. We further prove that in Transition-Independent MDPs (TI-MDPs), the observation-action history reduces to a tractable minimal local augmented state. However, we show through numerical experiments that although the optimal solution spaces are structurally isomorphic, the practical learning dynamics are fundamentally different. First, using the minimal local augmented state, the equivalence no longer holds when transitions are not independent. Second, operational constraints and causal credit-assignment errors in Temporal Difference (TD) algorithms induce different learning behaviors across regimes. Finally, leveraging this structural equivalence to bypass these learning challenges, we demonstrate successful multi-agent zero-shot policy transfer from OD to AD, paving the way for unified, efficient solution methods in complex delayed systems.

Related papers

• CoDiMAD: Diffusion-Based Privileged Distillation for Communication-Free Multi-Robot Coordination [2.17870369215002]
  CoDiMAD is a framework that trains a privileged oracle with MAPPO and distills the oracle into decentralized students.<n> Experiments on three cooperative tasks show that CoDiMAD consistently outperforms direct local MARL and deterministic distillation baselines.
  arXiv  Detail & Related papers  (2026-07-10T16:36:45Z)
• Unifying Temporal and Structural Credit Assignment in LLM-Based Multi-Agent Prompt Optimization [10.37712840622514]
  Multi-Agent Systems (MAS) empower Large Language Models to tackle complex reasoning computation tasks.<n>Existing black-boxs struggle to attribute trajectory-level failure to specific local components.<n>We argue that tractable MAS optimization needs structural inductive biases to disentangle error signals.
  arXiv  Detail & Related papers  (2026-05-28T16:57:57Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Emergent Coordination and Phase Structure in Independent Multi-Agent Reinforcement Learning [0.0]
  We run large-scale experiments across environment size L and agent density.<n>We construct a phase map using two axes - the cooperative success rate (CSR) and a stability index derived from TD-error variance.<n>We show that decentralized MARL exhibits a coherent phase structure governed by the interaction between scale, density, and kernel drift.
  arXiv  Detail & Related papers  (2025-11-28T16:14:31Z)
• Locality Preserving Markovian Transition for Instance Retrieval [59.16243976912006]
  We introduce the Locality Preserving Markovian Transition (LPMT) framework, which employs a long-term thermodynamic transition process with multiple states for accurate manifold distance measurement.<n>The proposed LPMT first integrates diffusion processes across separate graphs using Bidirectional Collaborative Diffusion (BCD) to establish strong similarity relationships.<n>After, Locality State Embedding (LSE) encodes each instance into a distribution for enhanced local consistency.<n>These distributions are interconnected via the Thermodynamic Markovian Transition (TMT) process, enabling efficient global retrieval while maintaining local effectiveness.
  arXiv  Detail & Related papers  (2025-06-05T16:07:31Z)
• Offline Multi-agent Reinforcement Learning via Score Decomposition [51.23590397383217]
  offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts.<n>This work is the first work to explicitly address the distributional gap between offline and online MARL.
  arXiv  Detail & Related papers  (2025-05-09T11:42:31Z)
• Locally Interdependent Multi-Agent MDP: Theoretical Framework for Decentralized Agents with Dynamic Dependencies [6.015898117103069]
  We analyze a decentralized model with dynamically varying dependencies called the Locally Interdependent Multi-Agent MDP. Despite the intractability that general partially observable multi-agent systems suffer from, we propose three closed-form policies.
  arXiv  Detail & Related papers  (2024-06-10T22:11:00Z)
• Strategic Distribution Shift of Interacting Agents via Coupled Gradient Flows [6.064702468344376]
  We propose a novel framework for analyzing the dynamics of distribution shift in real-world systems. We show that our approach captures well-documented forms of distribution shifts like polarization and disparate impacts that simpler models cannot capture.
  arXiv  Detail & Related papers  (2023-07-03T17:18:50Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• Causality-Based Multivariate Time Series Anomaly Detection [63.799474860969156]
  We formulate the anomaly detection problem from a causal perspective and view anomalies as instances that do not follow the regular causal mechanism to generate the multivariate data. We then propose a causality-based anomaly detection approach, which first learns the causal structure from data and then infers whether an instance is an anomaly relative to the local causal mechanism. We evaluate our approach with both simulated and public datasets as well as a case study on real-world AIOps applications.
  arXiv  Detail & Related papers  (2022-06-30T06:00:13Z)
• Provable RL with Exogenous Distractors via Multistep Inverse Dynamics [85.52408288789164]
  Real-world applications of reinforcement learning (RL) require the agent to deal with high-dimensional observations such as those generated from a megapixel camera. Prior work has addressed such problems with representation learning, through which the agent can provably extract endogenous, latent state information from raw observations. However, such approaches can fail in the presence of temporally correlated noise in the observations.
  arXiv  Detail & Related papers  (2021-10-17T15:21:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.