A reinforcement learning approach to rare trajectory sampling

• URL: http://arxiv.org/abs/2005.12890v3
• Date: Wed, 25 Nov 2020 15:14:23 GMT
• Title: A reinforcement learning approach to rare trajectory sampling
• Abstract summary: We present a general approach to adaptively construct a dynamics that efficiently samples atypical events. We do so by exploiting the methods of reinforcement learning (RL), which refers to the set of machine learning techniques aimed at finding the optimal behaviour. We discuss natural extensions of the ideas presented here, including to continuous-time Markov systems, first passage time problems and non-Markovian dynamics.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Very often when studying non-equilibrium systems one is interested in analysing dynamical behaviour that occurs with very low probability, so called rare events. In practice, since rare events are by definition atypical, they are often difficult to access in a statistically significant way. What are required are strategies to "make rare events typical" so that they can be generated on demand. Here we present such a general approach to adaptively construct a dynamics that efficiently samples atypical events. We do so by exploiting the methods of reinforcement learning (RL), which refers to the set of machine learning techniques aimed at finding the optimal behaviour to maximise a reward associated with the dynamics. We consider the general perspective of dynamical trajectory ensembles, whereby rare events are described in terms of ensemble reweighting. By minimising the distance between a reweighted ensemble and that of a suitably parametrised controlled dynamics we arrive at a set of methods similar to those of RL to numerically approximate the optimal dynamics that realises the rare behaviour of interest. As simple illustrations we consider in detail the problem of excursions of a random walker, for the case of rare events with a finite time horizon; and the problem of a studying current statistics of a particle hopping in a ring geometry, for the case of an infinite time horizon. We discuss natural extensions of the ideas presented here, including to continuous-time Markov systems, first passage time problems and non-Markovian dynamics.

Related papers

• Natural Invariant Measures for Chaotic Game Dynamics: Finding Order in Chaos [21.07105242653346]
  We study the long-term behavior of the Multiplicative Weights Update (MWU) algorithm in game settings.<n>We show that natural invariant measures provide the rigorous framework necessary to find order within this chaos.<n>Our results reveal that this simple learning algorithm captures the full spectrum of behaviors found in one-dimensional dynamical systems.
  arXiv  Detail & Related papers  (2026-07-23T20:45:08Z)
• Consistency Is Not Always Correct: Towards Understanding the Role of Exploration in Post-Training Reasoning [75.79451512757844]
  Foundation models exhibit broad knowledge but limited task-specific reasoning.<n> RLVR and inference scaling motivate post-training strategies such as RLVR and inference scaling.<n>We show that RLVR induces a squeezing effect, reducing reasoning entropy and forgetting some correct paths.
  arXiv  Detail & Related papers  (2025-11-10T18:25:26Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)
• Forecasting Continuous Non-Conservative Dynamical Systems in SO(3) [51.510040541600176]
  We propose a novel approach to modeling the rotation of moving objects in computer vision.<n>Our approach is agnostic to energy and momentum conservation while being robust to input noise.<n>By learning to approximate object dynamics from noisy states during training, our model attains robust extrapolation capabilities in simulation and various real-world settings.
  arXiv  Detail & Related papers  (2025-08-11T09:03:10Z)
• Whenever, Wherever: Towards Orchestrating Crowd Simulations with Spatio-Temporal Spawn Dynamics [65.72663487116439]
  We propose nTPP-GMM that models spawn-temporal spawn dynamics using Neural Temporal Point Processes. We evaluate our approach by simulations of three diverse real-world datasets with nTPP-GMM.
  arXiv  Detail & Related papers  (2025-03-20T18:46:41Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data. Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Decoupled Marked Temporal Point Process using Neural Ordinary Differential Equations [14.828081841581296]
  A Marked Temporal Point Process (MTPP) is a process whose realization is a set of event-time data. Recent studies have utilized deep neural networks to capture complex temporal dependencies of events. We propose a Decoupled MTPP framework that disentangles characterization of a process into a set of evolving influences from different events.
  arXiv  Detail & Related papers  (2024-06-10T10:15:32Z)
• Event Detection in Time Series: Universal Deep Learning Approach [0.0]
  Event detection in time series is a challenging task due to the prevalence of imbalanced datasets, rare events, and time interval-defined events. We propose a novel supervised regression-based deep learning approach that offers several advantages over classification-based methods. Our approach can effectively handle various types of events within a unified framework, including rare events and imbalanced datasets.
  arXiv  Detail & Related papers  (2023-11-27T09:33:56Z)
• Exact Inference for Continuous-Time Gaussian Process Dynamics [6.941863788146731]
  In practice, the true system is often unknown and has to be learned from measurement data. Most methods in Gaussian process (GP) dynamics model learning are trained on one-step ahead predictions. We show how to derive flexible inference schemes for these types of evaluations.
  arXiv  Detail & Related papers  (2023-09-05T16:07:00Z)
• Endogenous Macrodynamics in Algorithmic Recourse [52.87956177581998]
  Existing work on Counterfactual Explanations (CE) and Algorithmic Recourse (AR) has largely focused on single individuals in a static environment. We show that many of the existing methodologies can be collectively described by a generalized framework. We then argue that the existing framework does not account for a hidden external cost of recourse, that only reveals itself when studying the endogenous dynamics of recourse at the group level.
  arXiv  Detail & Related papers  (2023-08-16T07:36:58Z)
• Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation [66.86987509942607]
  We evaluate how such a paradigm should be done in imitation learning. We consider a setting where the pretraining corpus consists of multitask demonstrations. We argue that inverse dynamics modeling is well-suited to this setting.
  arXiv  Detail & Related papers  (2023-05-26T14:40:46Z)
• Finding the effective dynamics to make rare events typical in chaotic maps [0.0]
  We propose a framework for finding an effective topologically-conjugate map whose typical trajectories correspond to atypical ones of the original map. This work brings chaotic maps into the growing family of systems whose rare fluctuations can be characterized and controlled by means of a large-deviation formalism.
  arXiv  Detail & Related papers  (2023-04-26T18:00:08Z)
• Action Matching: Learning Stochastic Dynamics from Samples [10.46643972142224]
  Action Matching is a method for learning a rich family of dynamics using only independent samples from its time evolution. We derive a tractable training objective, which does not rely on explicit assumptions about the underlying dynamics. Inspired by connections with optimal transport, we derive extensions of Action Matching to learn differential equations and dynamics involving creation and destruction of probability mass.
  arXiv  Detail & Related papers  (2022-10-13T01:49:48Z)
• A Frequency Perspective of Adversarial Robustness [72.48178241090149]
  We present a frequency-based understanding of adversarial examples, supported by theoretical and empirical findings. Our analysis shows that adversarial examples are neither in high-frequency nor in low-frequency components, but are simply dataset dependent. We propose a frequency-based explanation for the commonly observed accuracy vs. robustness trade-off.
  arXiv  Detail & Related papers  (2021-10-26T19:12:34Z)
• Consistency of mechanistic causal discovery in continuous-time using Neural ODEs [85.7910042199734]
  We consider causal discovery in continuous-time for the study of dynamical systems. We propose a causal discovery algorithm based on penalized Neural ODEs.
  arXiv  Detail & Related papers  (2021-05-06T08:48:02Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)
• A Multi-Channel Neural Graphical Event Model with Negative Evidence [76.51278722190607]
  Event datasets are sequences of events of various types occurring irregularly over the time-line. We propose a non-parametric deep neural network approach in order to estimate the underlying intensity functions.
  arXiv  Detail & Related papers  (2020-02-21T23:10:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.