Interpretable Feature Engineering for Time Series Predictors using Attention Networks

• URL: http://arxiv.org/abs/2205.12723v1
• Date: Mon, 23 May 2022 20:13:08 GMT
• Title: Interpretable Feature Engineering for Time Series Predictors using Attention Networks
• Authors: Tianjie Wang, Jie Chen, Joel Vaughan, and Vijayan N. Nair
• Abstract summary: We use multi-head attention networks to develop interpretable features and use them to achieve good predictive performance. The customized attention layer explicitly uses multiplicative interactions and builds feature-engineering heads that capture temporal dynamics in a parsimonious manner.
• Score: 6.617546606897785
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Regression problems with time-series predictors are common in banking and many other areas of application. In this paper, we use multi-head attention networks to develop interpretable features and use them to achieve good predictive performance. The customized attention layer explicitly uses multiplicative interactions and builds feature-engineering heads that capture temporal dynamics in a parsimonious manner. Convolutional layers are used to combine multivariate time series. We also discuss methods for handling static covariates in the modeling process. Visualization and explanation tools are used to interpret the results and explain the relationship between the inputs and the extracted features. Both simulation and real dataset are used to illustrate the usefulness of the methodology. Keyword: Attention heads, Deep neural networks, Interpretable feature engineering

Related papers

• Contrastive Representation Learning for Dynamic Link Prediction in Temporal Networks [1.9389881806157312]
  We introduce a self-supervised method for learning representations of temporal networks. We propose a recurrent message-passing neural network architecture for modeling the information flow over time-respecting paths of temporal networks. The proposed method is tested on Enron, COLAB, and Facebook datasets.
  arXiv  Detail & Related papers  (2024-08-22T22:50:46Z)
• TimeGraphs: Graph-based Temporal Reasoning [64.18083371645956]
  TimeGraphs is a novel approach that characterizes dynamic interactions as a hierarchical temporal graph. Our approach models the interactions using a compact graph-based representation, enabling adaptive reasoning across diverse time scales. We evaluate TimeGraphs on multiple datasets with complex, dynamic agent interactions, including a football simulator, the Resistance game, and the MOMA human activity dataset.
  arXiv  Detail & Related papers  (2024-01-06T06:26:49Z)
• FocusLearn: Fully-Interpretable, High-Performance Modular Neural Networks for Time Series [0.3277163122167434]
  This paper proposes a novel modular neural network model for time series prediction that is interpretable by construction. A recurrent neural network learns the temporal dependencies in the data while an attention-based feature selection component selects the most relevant features. A modular deep network is trained from the selected features independently to show the users how features influence outcomes, making the model interpretable.
  arXiv  Detail & Related papers  (2023-11-28T14:51:06Z)
• Neural Koopman prior for data assimilation [7.875955593012905]
  We use a neural network architecture to embed dynamical systems in latent spaces. We introduce methods that enable to train such a model for long-term continuous reconstruction. The potential for self-supervised learning is also demonstrated, as we show the promising use of trained dynamical models as priors for variational data assimilation techniques.
  arXiv  Detail & Related papers  (2023-09-11T09:04:36Z)
• Explainable Parallel RCNN with Novel Feature Representation for Time Series Forecasting [0.0]
  Time series forecasting is a fundamental challenge in data science. We develop a parallel deep learning framework composed of RNN and CNN. Extensive experiments on three datasets reveal the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-05-08T17:20:13Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• Ripple: Concept-Based Interpretation for Raw Time Series Models in Education [5.374524134699487]
  Time series is the most prevalent form of input data for educational prediction tasks. We propose an approach that utilizes irregular multivariate time series modeling with graph neural networks to achieve comparable or better accuracy. We analyze these advances in the education domain, addressing the task of early student performance prediction.
  arXiv  Detail & Related papers  (2022-12-02T12:26:00Z)
• Learning the Evolutionary and Multi-scale Graph Structure for Multivariate Time Series Forecasting [50.901984244738806]
  We show how to model the evolutionary and multi-scale interactions of time series. In particular, we first provide a hierarchical graph structure cooperated with the dilated convolution to capture the scale-specific correlations. A unified neural network is provided to integrate the components above to get the final prediction.
  arXiv  Detail & Related papers  (2022-06-28T08:11:12Z)
• Self-Attention Neural Bag-of-Features [103.70855797025689]
  We build on the recently introduced 2D-Attention and reformulate the attention learning methodology. We propose a joint feature-temporal attention mechanism that learns a joint 2D attention mask highlighting relevant information.
  arXiv  Detail & Related papers  (2022-01-26T17:54:14Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks [91.65637773358347]
  We propose a general graph neural network framework designed specifically for multivariate time series data. Our approach automatically extracts the uni-directed relations among variables through a graph learning module. Our proposed model outperforms the state-of-the-art baseline methods on 3 of 4 benchmark datasets.
  arXiv  Detail & Related papers  (2020-05-24T04:02:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.