Predictable GRPO: A Closed-Form Model of Training Dynamics

• URL: http://arxiv.org/abs/2606.30789v2
• Date: Wed, 01 Jul 2026 14:28:25 GMT
• Title: Predictable GRPO: A Closed-Form Model of Training Dynamics
• Abstract summary: We develop a first-principles reduced-order model of these dynamics.<n>Group size enters, to leading order, as a noise temperature.<n>Across three models and two group sizes, the closed-form trajectory fits training reward to $R2 geq 0.91$.
• Score: 1.0537323885285492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We develop a first-principles reduced-order model of these dynamics. Under a single mean-field assumption that summarizes the policy by its expected reward, we reduce the GRPO update to a stochastically-forced damped oscillator whose mass, damping, and stiffness are fixed in closed form by the optimizer hyperparameters together with a single measured curvature scale -- momentum supplies the inertia, off-policy lag erodes the damping, and the group size enters, to leading order, as a noise temperature. The reduction has three consequences. First, it subsumes the empirical single-exponential saturation law as its overdamped limit, recasting the fitted plateau, timescale, and size exponent as the fixed point, inverse stiffness, and curvature-scaling exponent of the underlying potential, and adding, through the retained inertial term, the slow-start phase the single exponential cannot represent. Second, it yields predictions tied to independently measurable quantities rather than fitted ones: group-size invariance of the deterministic trajectory with a $1/G$ stationary fluctuation, a sharp stability threshold in the refresh interval, and an overdamped-to-oscillatory transition. Third, it furnishes diagnostics that separate failure modes a reward curve alone conflates -- reward hacking, advantage degeneracy, policy concentration, and dynamical instability. Across three models and two group sizes, the closed-form trajectory fits training reward to $R^2 \geq 0.91$ and the mean trajectory is group-size invariant to leading order -- on both the reward curve and out-of-distribution transfer to eight math benchmarks -- while the within-group reward spread retains a residual $G$-dependence that the leading-order temperature picture does not capture.

Related papers

• Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning [6.667218294299622]
  Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization.<n>Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios.<n>We propose the Entropy-Scaled Trust Region (ESTR), which scales each token's off-policy deviation by its local entropy.
  arXiv  Detail & Related papers  (2026-07-24T10:55:42Z)
• Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward [0.0]
  We study inverse reinforcement learning for mean-field games (MFGs) under an average-reward criterion.<n>We formulate the inverse problem by enforcing consistency with the expert mean-field term and long-run feature expectations.<n>Two numerical examples, a malware-spread MFG and an RKHS-based consumer-choice model, show that the recovered policies closely match expert behaviour.
  arXiv  Detail & Related papers  (2026-06-15T14:13:16Z)
• When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms [43.8784307709823]
  Long-horizon autoregressive forecasting of physical signals is limited by error accumulation.<n>We ask when such rollout stays stable using synthetic three-component seismograms as a physically structured testbed.
  arXiv  Detail & Related papers  (2026-06-09T13:46:14Z)
• A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training [0.0]
  This paper develops a mean-field theory for a single-layer causal multi-head self-attention model trained by cross-entropy minimization.<n>In the infinite-head limit, the averaged attention logits define a risk functional on probability measures, whose first variation generates a nonlinear Wasserstein gradient-flow equation.<n>We study the long-time behavior of the PDE: energy dissipation, convergence to the stationary set under compactness, convergence to a single stationary measure under topological or Kurdyka--ojasiewicz assumptions, and explicit convergence rates under gradient-domination conditions.
  arXiv  Detail & Related papers  (2026-06-09T06:38:27Z)
• The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training [48.37150760049573]
  We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early.<n>We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures.<n>We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.
  arXiv  Detail & Related papers  (2026-05-26T03:03:24Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models [56.67321805551389]
  Reinforcement learning (RL) has become an effective way to improve prompt alignment and perceptual quality in diffusion and flow-matching generators.<n>The sampler, controlling the exploration behavior and denoising dynamics, is thus part of the policy.<n>We propose a new sampler that balances effective exploration with stability.
  arXiv  Detail & Related papers  (2026-05-22T11:37:22Z)
• Recovering Physical Dynamics from Discrete Observations via Intrinsic Differential Consistency [12.266189649117003]
  We replace local supervision with a global structural constraint.<n>We train a time-conditioned secant velocity field whose deviation from this property serves two purposes.<n>As a training regularizer, it confines the hypothesis space to flows that compose consistently across temporal scales.
  arXiv  Detail & Related papers  (2026-05-08T20:24:32Z)
• Symmetric Equilibrium Propagation for Thermodynamic Diffusion Training [0.0]
  We show that a bilinearly-coupled analog substrate can physically realize dynamics at a projected three-to-four orders of magnitude energy advantage over digital inference.<n>We resolve whether the emphtraining loop can be closed on the same substrate without routing gradients through an external digital accelerator.
  arXiv  Detail & Related papers  (2026-04-26T17:05:05Z)
• A Nonasymptotic Theory of Gain-Dependent Error Dynamics in Behavior Cloning [4.822598110892847]
  Behavior cloning policies on position-controlled robots inherit the closed-loop response of the underlying PD controller.<n>We show that independent sub-Gaussian action errors propagate through the gain-dependent closed-loop dynamics.
  arXiv  Detail & Related papers  (2026-04-15T23:53:09Z)
• Momentum Further Constrains Sharpness at the Edge of Stochastic Stability [0.5405981353784005]
  Recent work suggests (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found.<n>We demonstrate that SGD with momentum exhibits an Edge of Stability (EoSS)-like regime with batch-size-dependent behavior.
  arXiv  Detail & Related papers  (2026-04-15T17:28:46Z)
• Non-Equilibrium Stochastic Dynamics as a Unified Framework for Insight and Repetitive Learning: A Kramers Escape Approach to Continual Learning [0.0]
  Continual learning in artificial neural networks is fundamentally limited by the stability-plasticity dilemma.<n>We show that both problems admit a common resolution within non-equilibrium statistical physics.
  arXiv  Detail & Related papers  (2026-04-05T15:42:23Z)
• A Generative Sampler for distributions with possible discrete parameter based on Reversibility [9.349846971147256]
  We propose a unified, target-gradient-free generative sampling framework applicable across diverse state spaces.<n>We minimize the Maximum Mean Discrepancy (MMD) between the joint distributions of forward and backward Markov trajectories.<n> Experiments show that our framework accurately reproduces thermodynamic observables and captures mode-switching behavior across all regimes.
  arXiv  Detail & Related papers  (2026-03-10T06:32:51Z)
• Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction [45.25461515976432]
  Plug-and-Play diffusion prior (DP) frameworks have emerged as a powerful paradigm for imaging reconstruction.<n>We present a novel approach to resolving bias-hallucination trade-off, achieving state-of-the-art gradients with significantly accelerated convergence.
  arXiv  Detail & Related papers  (2026-02-26T16:58:43Z)
• Forecasting as Rendering: A 2D Gaussian Splatting Framework for Time Series Forecasting [79.37674445572462]
  Time series forecasting (TSF) remains a challenging problem due to the intricate entanglement of intraperiod-fluctuations and interperiod-trends.<n>Treating reshaped tensors as static images results in a topological mismatch.<n> relying on uniform fixed-size representations allocates modeling capacity inefficiently.<n>TimeGS is a novel framework that fundamentally shifts the forecasting paradigm from regression to 2D generative rendering.
  arXiv  Detail & Related papers  (2026-02-10T14:13:36Z)
• Mutual Information Collapse Explains Disentanglement Failure in $β$-VAEs [4.155522769716163]
  $$-VAE is a framework for unsupervised disentanglement.<n> benchmarks such as MIG and SAP typically peak at intermediate $$ and collapse as regularization increases.<n>We introduce the $$-VAE, which decouples regularization pressure from informational collapse.
  arXiv  Detail & Related papers  (2026-02-09T23:38:11Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Broadening Target Distributions for Accelerated Diffusion Models via a Novel Analysis Approach [49.97755400231656]
  We show that a new accelerated DDPM sampler achieves accelerated performance for three broad distribution classes not considered before.<n>Our results show an improved dependency on the data dimension $d$ among accelerated DDPM type samplers.
  arXiv  Detail & Related papers  (2024-02-21T16:11:47Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.