Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces

• URL: http://arxiv.org/abs/2607.18554v1
• Date: Mon, 20 Jul 2026 22:41:43 GMT
• Title: Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces
• Abstract summary: We develop an algorithm for cooperative reinforcement learning in networked decision processes with continuous state and action spaces.<n>Each agent maintains a local actor over a graph neighborhood, and a localized temporal-difference critic evaluates a truncated actionvalue function.
• Score: 16.43184149906767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We develop the Continuous Distributed Coupled Policy Gradient (CDCPG) algorithm for cooperative reinforcement learning in networked Markov decision processes with continuous state and action spaces. Each agent maintains a local actor over a bounded graph neighborhood, and a localized least-squares temporal-difference critic evaluates a truncated action-value function through a spectral random-feature representation of the local transition kernel. The analysis makes four contributions. First, the truncated action-value function is constructed as a conditional expectation over the neighborhood, yielding a well-posed localized Bellman theory that removes the continuation-kernel mismatch of naive truncation arguments. Second, we expose a dimensional obstruction to temporal-difference stability for normalized random features and prove an unconditional excitation bound that reduces stability to a symmetric persistence-of-excitation condition, monitorable through an online matrix-concentration certificate. Third, under exponential spatial decay of agent interactions, the excitation condition, and smoothness of the objective, CDCPG drives an averaged per-agent stationarity measure to within any excess $ε$ of an explicitly characterized approximation floor using $\widetilde{\mathcal{O}}(ε^{-2})$ shared-oracle samples, and the excess dependence matches the smooth nonconvex first-order rate; per-agent computation and communication are governed by the neighborhood size rather than the network size. Fourth, an adaptive-locality rule selects the radius that balances truncation and graph-decay residuals against the target accuracy. Experiments on a networked linear-quadratic benchmark corroborate the locality and feature-dimension predictions.

Related papers

• Spectral Dynamics of Semantic Drift in Clinical Multi-Agent Language Model Networks [0.0]
  We show that localized dense cliques confine hallucinated data, preventing global consensus and forcing the system toward a permanent entropy saturation threshold.<n>By introducing dynamic spectral monitoring operating at an $mathcalO(N3)$ time complexity, we present a mathematically rigorous technique to ensure global state diffusion.
  arXiv  Detail & Related papers  (2026-07-23T20:20:45Z)
• Radial Interaction Tomography: Recognizing Non-Transitive Evolutionary Games from One Range-Expansion Image [0.2864713389096699]
  We formulate a computer-vision inverse problem: from one endpoint image of an accretive multi-type expansion, recover the radius-indexed pairwise boundary-flow field.<n>The observable is a geometric signal extracted from sector-boundary curves in log-polar coordinates.
  arXiv  Detail & Related papers  (2026-07-01T03:23:47Z)
• Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning [81.57028614960576]
  We show that the Confidence Region possesses two critical properties: 1) High Reliability -- answers in the confidence region become highly accurate and stable, and 2) High Redundancy -- models generate unnecessary tokens long after reaching the correct answer.<n>These properties unlock more efficient and reliable inference strategies.
  arXiv  Detail & Related papers  (2026-06-01T10:11:14Z)
• Learning Transferable Predictability Representations [2.1505812656718795]
  We study the problem of assigning a scalar score to a short trajectory window that reflects its position on an ordered continuum of predictability regimes.<n>We formalize this as ordinal estimation over a five-level predictability ladder and identify a structural source of cross-system ambiguity.<n>We propose the Gauge-Fixed Ordinal Network (GON), a temporal convolutional model trained with an anchor-and-variance objective that pins level-wise score means to shared target coordinates.
  arXiv  Detail & Related papers  (2026-05-28T21:38:49Z)
• Recovering Physical Dynamics from Discrete Observations via Intrinsic Differential Consistency [12.266189649117003]
  We replace local supervision with a global structural constraint.<n>We train a time-conditioned secant velocity field whose deviation from this property serves two purposes.<n>As a training regularizer, it confines the hypothesis space to flows that compose consistently across temporal scales.
  arXiv  Detail & Related papers  (2026-05-08T20:24:32Z)
• Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RL [8.809468023364703]
  Design of robust reinforcement learning algorithms poses fundamental challenges.<n>We propose an approximate DRRL framework based on a first-order expansion of the relevant robust functional.<n>To learn the fixed point of this approximate equation, we propose Mean-Variance Approximation (MVSA)
  arXiv  Detail & Related papers  (2026-05-08T19:24:28Z)
• Decentralized Proximal Stochastic Gradient Langevin Dynamics [4.385194124090593]
  Decentralized Proximal Gradient Langevin Dynamics (DE-PSGLD) for sampling from a log-concave probability distribution constrained to a convex domain.<n>As the first decentralized approach for constrained domains, our algorithm exhibits fast posterior concentration and high predictive accuracy.
  arXiv  Detail & Related papers  (2026-05-01T15:11:06Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation [46.48569176651487]
  We prove a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data.<n>We extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications.
  arXiv  Detail & Related papers  (2024-05-31T17:36:16Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• On the Convergence of Stochastic Extragradient for Bilinear Games with Restarted Iteration Averaging [96.13485146617322]
  We present an analysis of the ExtraGradient (SEG) method with constant step size, and present variations of the method that yield favorable convergence. We prove that when augmented with averaging, SEG provably converges to the Nash equilibrium, and such a rate is provably accelerated by incorporating a scheduled restarting procedure.
  arXiv  Detail & Related papers  (2021-06-30T17:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.