Deep Fourier Kernel for Self-Attentive Point Processes

• URL: http://arxiv.org/abs/2002.07281v5
• Date: Sun, 21 Feb 2021 05:33:59 GMT
• Title: Deep Fourier Kernel for Self-Attentive Point Processes
• Authors: Shixiang Zhu, Minghe Zhang, Ruyi Ding, Yao Xie
• Abstract summary: We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We introduce a novel score function using Fourier kernel embedding, whose spectrum is represented using neural networks. We establish our approach's theoretical properties and demonstrate our approach's competitive performance compared to the state-of-the-art for synthetic and real data.
• Score: 16.63706478353667
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We borrow the idea from the attention mechanism and incorporate it into the point processes' conditional intensity function. We further introduce a novel score function using Fourier kernel embedding, whose spectrum is represented using neural networks, which drastically differs from the traditional dot-product kernel and can capture a more complex similarity structure. We establish our approach's theoretical properties and demonstrate our approach's competitive performance compared to the state-of-the-art for synthetic and real data.

Related papers

• Nonstationary Sparse Spectral Permanental Process [24.10531062895964]
  We propose a novel approach utilizing the sparse spectral representation of nonstationary kernels. This technique relaxes the constraints on kernel types and stationarity, allowing for more flexible modeling. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-10-04T16:40:56Z)
• Injecting Hamiltonian Architectural Bias into Deep Graph Networks for Long-Range Propagation [55.227976642410766]
  dynamics of information diffusion within graphs is a critical open issue that heavily influences graph representation learning. Motivated by this, we introduce (port-)Hamiltonian Deep Graph Networks. We reconcile under a single theoretical and practical framework both non-dissipative long-range propagation and non-conservative behaviors.
  arXiv  Detail & Related papers  (2024-05-27T13:36:50Z)
• Complex Recurrent Spectral Network [1.0499611180329806]
  This paper presents a novel approach to advancing artificial intelligence (AI) through the development of the Complex Recurrent Spectral Network ($mathbbC$-RSN) The $mathbbC$-RSN is designed to address a critical limitation in existing neural network models: their inability to emulate the complex processes of biological neural networks.
  arXiv  Detail & Related papers  (2023-12-12T14:14:40Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Linear Self-Attention Approximation via Trainable Feedforward Kernel [77.34726150561087]
  In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches. We aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.
  arXiv  Detail & Related papers  (2022-11-08T08:14:11Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• Neural Spectral Marked Point Processes [18.507050473968985]
  We introduce a novel and general neural network-based non-stationary influence kernel for handling complex discrete events. We demonstrate the superior performance of our proposed method compared with the state-of-the-art on synthetic and real data.
  arXiv  Detail & Related papers  (2021-06-20T23:00:37Z)
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information [18.204325860752768]
  Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes. We provide a principled way to characterize continuous-time systems using deep learning tools.
  arXiv  Detail & Related papers  (2021-03-28T20:13:53Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)
• Convolutional Spectral Kernel Learning [21.595130250234646]
  We build an interpretable convolutional spectral kernel network (textttCSKN) based on the inverse Fourier transform. We derive the generalization error bounds and introduce two regularizers to improve the performance. Experiments results on real-world datasets validate the effectiveness of the learning framework.
  arXiv  Detail & Related papers  (2020-02-28T14:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.