A Temporal Kernel Approach for Deep Learning with Continuous-time Information

• URL: http://arxiv.org/abs/2103.15213v1
• Date: Sun, 28 Mar 2021 20:13:53 GMT
• Title: A Temporal Kernel Approach for Deep Learning with Continuous-time Information
• Authors: Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, Kannan Achan
• Abstract summary: Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes. We provide a principled way to characterize continuous-time systems using deep learning tools.
• Score: 18.204325860752768
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes. Current approaches often handle time in a heuristic manner to be consistent with the existing deep learning architectures and implementations. In this paper, we provide a principled way to characterize continuous-time systems using deep learning tools. Notably, the proposed approach applies to all the major deep learning architectures and requires little modifications to the implementation. The critical insight is to represent the continuous-time system by composing neural networks with a temporal kernel, where we gain our intuition from the recent advancements in understanding deep learning with Gaussian process and neural tangent kernel. To represent the temporal kernel, we introduce the random feature approach and convert the kernel learning problem to spectral density estimation under reparameterization. We further prove the convergence and consistency results even when the temporal kernel is non-stationary, and the spectral density is misspecified. The simulations and real-data experiments demonstrate the empirical effectiveness of our temporal kernel approach in a broad range of settings.

Related papers

• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Self-STORM: Deep Unrolled Self-Supervised Learning for Super-Resolution Microscopy [55.2480439325792]
  We introduce deep unrolled self-supervised learning, which alleviates the need for such data by training a sequence-specific, model-based autoencoder. Our proposed method exceeds the performance of its supervised counterparts.
  arXiv  Detail & Related papers  (2024-03-25T17:40:32Z)
• Spatio-temporal point processes with deep non-stationary kernels [18.10670233156497]
  We develop a new deep non-stationary influence kernel that can model non-stationary-temporal point processes. The main idea is to approximate the influence kernel with a novel and general low-rank decomposition. We also take a new approach to maintain the non-negativity constraint of the conditional intensity by introducing a log-barrier penalty.
  arXiv  Detail & Related papers  (2022-11-21T04:49:39Z)
• Online Attentive Kernel-Based Temporal Difference Learning [13.94346725929798]
  Online Reinforcement Learning (RL) has been receiving increasing attention due to its fast learning capability and improving data efficiency. Online RL often suffers from complex Value Function Approximation (VFA) and catastrophic interference. We propose an Online Attentive Kernel-Based Temporal Difference (OAKTD) algorithm using two-timescale optimization.
  arXiv  Detail & Related papers  (2022-01-22T14:47:10Z)
• Neural Point Process for Learning Spatiotemporal Event Dynamics [21.43984242938217]
  We propose a deep dynamics model that integratestemporal point processes. Our method is flexible, efficient and can accurately forecast irregularly sampled events over space and time. On real-world benchmarks, our model demonstrates superior performance over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-12-12T23:17:33Z)
• Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series [8.918419734720613]
  A simple and interpretable way to learn a dynamical system from data is to interpolate its vector-field with a kernel. Despite its previous successes, this strategy breaks down when the observed time series is not regularly sampled in time. We propose to address this problem by directly approxing the vector field of the dynamical system by incorporating time differences in the (KF) data-adapted kernels.
  arXiv  Detail & Related papers  (2021-11-25T11:45:40Z)
• Consistency of mechanistic causal discovery in continuous-time using Neural ODEs [85.7910042199734]
  We consider causal discovery in continuous-time for the study of dynamical systems. We propose a causal discovery algorithm based on penalized Neural ODEs.
  arXiv  Detail & Related papers  (2021-05-06T08:48:02Z)
• Neural ODE Processes [64.10282200111983]
  We introduce Neural ODE Processes (NDPs), a new class of processes determined by a distribution over Neural ODEs. We show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.
  arXiv  Detail & Related papers  (2021-03-23T09:32:06Z)
• Contrastive learning of strong-mixing continuous-time stochastic processes [53.82893653745542]
  Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data. We show that a properly constructed contrastive learning task can be used to estimate the transition kernel for small-to-mid-range intervals in the diffusion case.
  arXiv  Detail & Related papers  (2021-03-03T23:06:47Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)
• Deep Fourier Kernel for Self-Attentive Point Processes [16.63706478353667]
  We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We introduce a novel score function using Fourier kernel embedding, whose spectrum is represented using neural networks. We establish our approach's theoretical properties and demonstrate our approach's competitive performance compared to the state-of-the-art for synthetic and real data.
  arXiv  Detail & Related papers  (2020-02-17T22:25:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.