Non-adversarial training of Neural SDEs with signature kernel scores

• URL: http://arxiv.org/abs/2305.16274v1
• Date: Thu, 25 May 2023 17:31:18 GMT
• Title: Non-adversarial training of Neural SDEs with signature kernel scores
• Authors: Zacharia Issa and Blanka Horvath and Maud Lemercier and Cristopher Salvi
• Abstract summary: State-of-the-art performance for irregular time series generation has been previously obtained by training these models adversarially as GANs. In this paper, we introduce a novel class of scoring rules on pathspace based on signature kernels.
• Score: 4.721845865189578
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural SDEs are continuous-time generative models for sequential data. State-of-the-art performance for irregular time series generation has been previously obtained by training these models adversarially as GANs. However, as typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty to mitigate these issues. In this paper, we introduce a novel class of scoring rules on pathspace based on signature kernels and use them as objective for training Neural SDEs non-adversarially. By showing strict properness of such kernel scores and consistency of the corresponding estimators, we provide existence and uniqueness guarantees for the minimiser. With this formulation, evaluating the generator-discriminator pair amounts to solving a system of linear path-dependent PDEs which allows for memory-efficient adjoint-based backpropagation. Moreover, because the proposed kernel scores are well-defined for paths with values in infinite dimensional spaces of functions, our framework can be easily extended to generate spatiotemporal data. Our procedure permits conditioning on a rich variety of market conditions and significantly outperforms alternative ways of training Neural SDEs on a variety of tasks including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.

Related papers

• Trajectory Flow Matching with Applications to Clinical Time Series Modeling [77.58277281319253]
  Trajectory Flow Matching (TFM) trains a Neural SDE in a simulation-free manner, bypassing backpropagation through the dynamics. We demonstrate improved performance on three clinical time series datasets in terms of absolute performance and uncertainty prediction.
  arXiv  Detail & Related papers  (2024-10-28T15:54:50Z)
• Efficient Training of Neural Stochastic Differential Equations by Matching Finite Dimensional Distributions [3.889230974713832]
  We develop a novel scoring rule for comparing continuous Markov processes. This scoring rule allows us to bypass the computational overhead associated with signature kernels. We demonstrate that FDM achieves superior performance, consistently outperforming existing methods in terms of both computational efficiency and generative quality.
  arXiv  Detail & Related papers  (2024-10-04T23:26:38Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data [3.686808512438363]
  Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods. We propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.
  arXiv  Detail & Related papers  (2024-02-22T22:00:03Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction with Exogenous Variables [22.145726318053526]
  Inter-series correlation and time dependence among variables are rarely considered in the present continuous methods. We propose a continuous-time model for arbitrary-step prediction to learn an unknown PDE system.
  arXiv  Detail & Related papers  (2022-08-03T08:34:31Z)
• Closed-form Continuous-Depth Models [99.40335716948101]
  Continuous-depth neural models rely on advanced numerical differential equation solvers. We present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster.
  arXiv  Detail & Related papers  (2021-06-25T22:08:51Z)
• Neural SDEs as Infinite-Dimensional GANs [18.07683058213448]
  We show that the current classical approach to fitting SDEs may be approached as a special case of (Wasserstein) GANs. We obtain Neural SDEs as (in modern machine learning parlance) continuous-time generative time series models.
  arXiv  Detail & Related papers  (2021-02-06T19:59:15Z)
• STENCIL-NET: Data-driven solution-adaptive discretization of partial differential equations [2.362412515574206]
  We present STENCIL-NET, an artificial neural network architecture for data-driven learning of problem- and resolution-specific local discretizations of nonlinear PDEs. Knowing the actual PDE is not necessary, as solution data is sufficient to train the network to learn the discrete operators. A once-trained STENCIL-NET model can be used to predict solutions of the PDE on larger domains and for longer times than it was trained for.
  arXiv  Detail & Related papers  (2021-01-15T15:43:41Z)
• Probabilistic Circuits for Variational Inference in Discrete Graphical Models [101.28528515775842]
  Inference in discrete graphical models with variational methods is difficult. Many sampling-based methods have been proposed for estimating Evidence Lower Bound (ELBO) We propose a new approach that leverages the tractability of probabilistic circuit models, such as Sum Product Networks (SPN) We show that selective-SPNs are suitable as an expressive variational distribution, and prove that when the log-density of the target model is aweighted the corresponding ELBO can be computed analytically.
  arXiv  Detail & Related papers  (2020-10-22T05:04:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.