On Distributional Reinforcement Learning in Chaotic Dynamical Systems

• URL: http://arxiv.org/abs/2605.30160v1
• Date: Thu, 28 May 2026 16:17:32 GMT
• Title: On Distributional Reinforcement Learning in Chaotic Dynamical Systems
• Abstract summary: Chaotic dynamical systems pose a fundamental challenge for Reinforcement Learning (RL)<n>We show that under mild statistical stability assumptions, the return distribution evolves more regularly than individual trajectories when measured under the $1$-Wasserstein metric.<n>We offer a principled explanation for the advantages of distributional methods in chaotic systems and the optimisation of RL objectives under chaos.
• Score: 6.932921577765093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chaotic dynamical systems pose a fundamental challenge for Reinforcement Learning (RL): exponential sensitivity to initial conditions induces high-variance bootstrap targets and poorly conditioned gradient updates. Chaotic dynamics arise across scientific and engineering domains, from fluid flows and climate systems to multi-agent systems, where reliable learning is highly desirable. Standard RL methods optimise expected returns through scalar value functions, implicitly averaging over diverging trajectories and entangling trajectory level instability with the learning objective. We show that under mild statistical stability assumptions, the return distribution evolves more regularly than individual trajectories when measured under the $1$-Wasserstein metric, yielding a smoother distributional Bellman objective. By aligning optimisation with this measure level structure, distributional RL provides better conditioned learning. We offer a principled explanation for the advantages of distributional methods in chaotic systems and the geometries of RL objectives under chaos.

Related papers

• S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning [1.160208922584163]
  We study whether subgoal selection by the high-level agent can be performed more strategically, by providing it with dynamics-aware intrinsic motivation.<n>This approach stabilizes the high-level policy by learning to minimize the predictive uncertainty associated with the coarse dynamics.<n> Empirically, we observe that a dense, dynamics-aware intrinsic reward leads to risk-averse subgoal selection, enabling it to outperform state-of-the-art HRL methods in non-stationary long-horizon environments.
  arXiv  Detail & Related papers  (2026-07-21T16:03:34Z)
• Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems [0.3499870393443268]
  Reinforcement learning (RL) has emerged as a promising feedback control strategy for nonlinear and complex dynamical systems.<n>In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm.
  arXiv  Detail & Related papers  (2026-07-17T17:56:05Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving [22.3805998088591]
  DACER-F is a flow matching algorithm for generative policies in autonomous driving systems.<n>It achieves a score of 775.8 on the humanoid-stand task and surpassing prior methods.
  arXiv  Detail & Related papers  (2026-03-03T05:35:53Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Model-Agnostic Solutions for Deep Reinforcement Learning in Non-Ergodic Contexts [3.5577285720638194]
  Reinforcement Learning (RL) remains a central optimisation framework in machine learning.<n>Bellman equation, central to most RL algorithms, is formulated in terms of expected values of future rewards.<n>In non-ergodic environments, the ensemble average diverges from the time-average growth experienced by individual agents.
  arXiv  Detail & Related papers  (2026-01-13T16:53:40Z)
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning [49.60927398960061]
  We propose a neuro-symbolic Deep RL approach that integrates background symbolic knowledge to improve sample efficiency.<n>Online reasoning is performed to guide the training process through two mechanisms.<n>We show improved performance over a state-of-the-art reward machine baseline.
  arXiv  Detail & Related papers  (2026-01-06T09:28:53Z)
• Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models [0.0]
  We propose Dynamic Rank Reinforcement Learning (DR-RL), a novel framework that adaptively optimize the low-rank factorization of Multi-Head Self-Attention (MHSA) in Large Language Models (LLMs)<n>DR-RL maintains downstream accuracy statistically equivalent to full-rank attention while significantly reducing Floating Point Operations (FLOPs)<n>This work bridges the gap between adaptive efficiency and theoretical rigor in MHSA, offering a principled, mathematically grounded alternative to rank reduction techniques in resource-constrained deep learning.
  arXiv  Detail & Related papers  (2025-12-17T21:09:19Z)
• Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones [3.7312377768685714]
  We propose Adaptive Trajectory Policy Optimization (ATPO), a lightweight step-selection strategy that dynamically reallocates gradient updates to high-leverage steps without changing the RL objective, rewards, or compute budget.<n>ATPO delivers substantial gains in reasoning accuracy and training stability across benchmarks, showing that exploiting trajectory dynamics is key to advancing dLLM RL.
  arXiv  Detail & Related papers  (2025-11-19T07:59:34Z)
• On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning [59.11784194183928]
  Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs)<n>Regularized Policy Gradient (RPG) view shows that the widely-used $k_3$ penalty is exactly the unnormalized KL.<n>RPG-REINFORCE with RPG-Style Clip improves accuracy by up to $+6$ absolute percentage points over DAPO.
  arXiv  Detail & Related papers  (2025-05-23T06:01:21Z)
• ROCM: RLHF on consistency models [8.905375742101707]
  We propose a reward optimization framework for applying RLHF to consistency models.<n>We investigate various $f$-divergences as regularization strategies, striking a balance between reward and model consistency.
  arXiv  Detail & Related papers  (2025-03-08T11:19:48Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.