XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation

• URL: http://arxiv.org/abs/2506.02694v1
• Date: Tue, 03 Jun 2025 09:43:45 GMT
• Title: XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation
• Abstract summary: We propose a novel attention mechanism based on Chatterjee's rank correlation coefficient.<n>We replace the matrix multiplication in standard attention mechanisms with this rank coefficient to measure the query-key relationship.<n>Our proposed mechanism, XicorAttention,'' integrates it into several state-of-the-art Transformer models.
• Score: 20.956872958560346
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Various Transformer-based models have been proposed for time series forecasting. These models leverage the self-attention mechanism to capture long-term temporal or variate dependencies in sequences. Existing methods can be divided into two approaches: (1) reducing computational cost of attention by making the calculations sparse, and (2) reshaping the input data to aggregate temporal features. However, existing attention mechanisms may not adequately capture inherent nonlinear dependencies present in time series data, leaving room for improvement. In this study, we propose a novel attention mechanism based on Chatterjee's rank correlation coefficient, which measures nonlinear dependencies between variables. Specifically, we replace the matrix multiplication in standard attention mechanisms with this rank coefficient to measure the query-key relationship. Since computing Chatterjee's correlation coefficient involves sorting and ranking operations, we introduce a differentiable approximation employing SoftSort and SoftRank. Our proposed mechanism, ``XicorAttention,'' integrates it into several state-of-the-art Transformer models. Experimental results on real-world datasets demonstrate that incorporating nonlinear correlation into the attention improves forecasting accuracy by up to approximately 9.1\% compared to existing models.

Related papers

• Extraction of informative statistical features in the problem of forecasting time series generated by It{ô}-type processes [65.01025489527173]
  We use the parameters of statistically adjusted mixture-type models of the observed regularities of the behavior of the time series.<n>We show that the use of additional statistical features improves the prediction.
  arXiv  Detail & Related papers  (2026-04-18T06:31:21Z)
• Relational Conformal Prediction for Correlated Time Series [56.59852921638328]
  We propose a novel distribution-free approach based on conformal prediction framework and quantile regression.<n>We fill this void by introducing a novel conformal prediction method based on graph deep learning operators.<n>Our approach provides accurate coverage and archives state-of-the-art uncertainty quantification in relevant benchmarks.
  arXiv  Detail & Related papers  (2025-02-13T16:12:17Z)
• Multivariate Probabilistic Time Series Forecasting with Correlated Errors [17.212396544233307]
  We introduce a plug-and-play method that learns the covariance structure of errors over multiple steps for autoregressive models. We evaluate our method on probabilistic models built on RNNs and Transformer architectures.
  arXiv  Detail & Related papers  (2024-02-01T20:27:19Z)
• Deep Ensembles Meets Quantile Regression: Uncertainty-aware Imputation for Time Series [45.76310830281876]
  We propose Quantile Sub-Ensembles, a novel method to estimate uncertainty with ensemble of quantile-regression-based task networks. Our method not only produces accurate imputations that is robust to high missing rates, but also is computationally efficient due to the fast training of its non-generative model.
  arXiv  Detail & Related papers  (2023-12-03T05:52:30Z)
• Correlated Attention in Transformers for Multivariate Time Series [22.542109523780333]
  We propose a novel correlated attention mechanism, which efficiently captures feature-wise dependencies, and can be seamlessly integrated within the encoder blocks of existing Transformers. In particular, correlated attention operates across feature channels to compute cross-covariance matrices between queries and keys with different lag values, and selectively aggregate representations at the sub-series level. This architecture facilitates automated discovery and representation learning of not only instantaneous but also lagged cross-correlations, while inherently capturing time series auto-correlation.
  arXiv  Detail & Related papers  (2023-11-20T17:35:44Z)
• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Efficient Interpretable Nonlinear Modeling for Multiple Time Series [5.448070998907116]
  This paper proposes an efficient nonlinear modeling approach for multiple time series. It incorporates nonlinear interactions among different time-series variables. Experimental results show that the proposed algorithm improves the identification of the support of the VAR coefficients in a parsimonious manner.
  arXiv  Detail & Related papers  (2023-09-29T11:42:59Z)
• TACTiS: Transformer-Attentional Copulas for Time Series [76.71406465526454]
  estimation of time-varying quantities is a fundamental component of decision making in fields such as healthcare and finance. We propose a versatile method that estimates joint distributions using an attention-based decoder. We show that our model produces state-of-the-art predictions on several real-world datasets.
  arXiv  Detail & Related papers  (2022-02-07T21:37:29Z)
• Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences [52.6022911513076]
  Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. We propose Linformer and Informer to reduce the quadratic complexity to linear (modulo logarithmic factors) via low-dimensional projection and row selection. Based on the theoretical analysis, we propose Skeinformer to accelerate self-attention and further improve the accuracy of matrix approximation to self-attention.
  arXiv  Detail & Related papers  (2021-12-10T06:58:05Z)
• Scalable Intervention Target Estimation in Linear Models [52.60799340056917]
  Current approaches to causal structure learning either work with known intervention targets or use hypothesis testing to discover the unknown intervention targets. This paper proposes a scalable and efficient algorithm that consistently identifies all intervention targets. The proposed algorithm can be used to also update a given observational Markov equivalence class into the interventional Markov equivalence class.
  arXiv  Detail & Related papers  (2021-11-15T03:16:56Z)
• Observation Error Covariance Specification in Dynamical Systems for Data assimilation using Recurrent Neural Networks [0.5330240017302621]
  We propose a data-driven approach based on long short term memory (LSTM) recurrent neural networks (RNN) The proposed approach does not require any knowledge or assumption about prior error distribution. We have compared the novel approach with two state-of-the-art covariance tuning algorithms, namely DI01 and D05.
  arXiv  Detail & Related papers  (2021-11-11T20:23:00Z)
• A Non-linear Function-on-Function Model for Regression with Time Series Data [11.738565299608721]
  We propose a general functional mapping that embraces the function-on-function linear model as a special case. We then propose a non-linear function-on-function model using the fully connected neural network to learn the mapping from data. The effectiveness of the proposed model is demonstrated through the application to two real-world problems.
  arXiv  Detail & Related papers  (2020-11-24T20:51:27Z)
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation [77.7420231319632]
  We adapt contextual generation of categorical sequences to a policy gradient estimator, which evaluates a set of correlated Monte Carlo (MC) rollouts for variance control. We also demonstrate the use of correlated MC rollouts for binary-tree softmax models, which reduce the high generation cost in large vocabulary scenarios.
  arXiv  Detail & Related papers  (2019-12-31T03:01:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.