TwinFormer: A Dual-Level Transformer for Long-Sequence Time-Series Forecasting

• URL: http://arxiv.org/abs/2512.12301v1
• Date: Sat, 13 Dec 2025 11:50:18 GMT
• Title: TwinFormer: A Dual-Level Transformer for Long-Sequence Time-Series Forecasting
• Abstract summary: TwinFormer is a hierarchical Transformer for long-sequence time-series forecasting.<n>It divides the input into non-overlapping temporal patches and processes them in two stages.<n>A lightweight GRU aggregates the globally contextualized patch tokens for direct multi-horizon prediction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: TwinFormer is a hierarchical Transformer for long-sequence time-series forecasting. It divides the input into non-overlapping temporal patches and processes them in two stages: (1) a Local Informer with top-$k$ Sparse Attention models intra-patch dynamics, followed by mean pooling; (2) a Global Informer captures long-range inter-patch dependencies using the same top-$k$ attention. A lightweight GRU aggregates the globally contextualized patch tokens for direct multi-horizon prediction. The resulting architecture achieves linear $O(kLd)$ time and memory complexity. On eight real-world benchmarking datasets from six different domains, including weather, stock price, temperature, power consumption, electricity, and disease, and forecasting horizons $96-720$, TwinFormer secures $27$ positions in the top two out of $34$. Out of the $27$, it achieves the best performance on MAE and RMSE at $17$ places and $10$ at the second-best place on MAE and RMSE. This consistently outperforms PatchTST, iTransformer, FEDformer, Informer, and vanilla Transformers. Ablations confirm the superiority of top-$k$ Sparse Attention over ProbSparse and the effectiveness of GRU-based aggregation. Code is available at this repository: https://github.com/Mahimakumavat1205/TwinFormer.

Related papers

• TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes [6.460910545652521]
  TEMPO is a makespan-aware dispatcher solving per-batch dispatch in milliseconds off the critical path.<n>Its SG integration runs out-of-process and fuses dispatch with count into one in-graph kernel.<n>End-to-end on Testbed B, Qwen3-235B (inside the win region) gains $4$--$6%$ throughput and cuts p99 latency by $sim 15.6%$; DeepSeek-V3 shows only mechanism cost.
  arXiv  Detail & Related papers  (2026-08-13T10:21:11Z)
• How Good Can Linear Models Be for Time-Series Forecasting? [1.558068541168739]
  We use Ridge regression as the testbed, since it has a closed-form solution and interpretable weights.<n>We search over context length, local normalization, regularization, and augmentation on eight standard benchmarks.<n>The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, CNN baselines on six of eight benchmarks.
  arXiv  Detail & Related papers  (2026-06-25T16:57:50Z)
• H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers [124.11648300910444]
  We present a hierarchical plug-and-play pruning-and-$-recovering framework, called Hierarchical Hourglass Tokenizer (H$_2$OT)<n>Our method is general-purpose: it can be easily incorporated into common VPT models on both seq2seq and seq2frame pipelines.
  arXiv  Detail & Related papers  (2025-09-08T17:59:59Z)
• Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling [50.994194925685434]
  LrcSSM is a $textitnon-linear$ recurrent model that processes long sequences as fast as today's linear state-space layers.<n>By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel.<n>LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 do not provide.
  arXiv  Detail & Related papers  (2025-05-27T20:02:59Z)
• SWIFT: Mapping Sub-series with Wavelet Decomposition Improves Time Series Forecasting [2.6764607949560593]
  $textitSWIFT$ is a lightweight model that is powerful, but also efficient in deployment and inference for Long-term Time Series Forecasting.<n>We conduct comprehensive experiments, and the results show that $textitSWIFT$ achieves state-of-the-art (SOTA) performance on multiple datasets.
  arXiv  Detail & Related papers  (2025-01-27T16:26:07Z)
• Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting [12.103678233732584]
  We propose a new Transformer, Sensorformer, which first compresses the global patch information and then simultaneously extracts cross-variable and cross-time dependencies from the compressed representations.<n>Sensorformer can effectively capture the correct inter-variable correlations and causal relationships, even in the presence of dynamic causal lags between variables.
  arXiv  Detail & Related papers  (2025-01-06T03:14:47Z)
• Nimbus: Secure and Efficient Two-Party Inference for Transformers [26.548997436676295]
  This work presents a new two-party inference framework $mathsfNimbus$ for Transformer models. For the linear layer, we propose a new 2PC paradigm along with an encoding approach to securely compute matrix multiplications based on an outer-product insight. For the non-linear layer, we propose an approach of low-degree approximation for $mathsfGELU$ and $mathsfSoftmax$.
  arXiv  Detail & Related papers  (2024-11-24T04:24:31Z)
• Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval [49.825549809652436]
  $k$NN-MT constructs an external datastore to store domain-specific translation knowledge. adaptive retrieval ($k$NN-MT-AR) dynamically estimates $lambda$ and skips $k$NN retrieval if $lambda$ is less than a fixed threshold. We propose dynamic retrieval ($k$NN-MT-DR) that significantly extends vanilla $k$NN-MT in two aspects.
  arXiv  Detail & Related papers  (2024-06-10T07:36:55Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing [52.64837396100988]
  MEGA is a recent transformer-based architecture, which utilizes a linear recurrent operator whose parallel computation, based on the FFT, scales as $O(LlogL)$, with $L$ being the sequence length. We build upon their approach by replacing the linear recurrence with a special temporal convolutional network which permits larger receptive field size with shallower networks, and reduces the computational complexity to $O(L)$. We evaluate TCNCA on EnWik8 language modeling, long-range-arena (LRA) sequence classification, as well as a synthetic reasoning benchmark associative recall.
  arXiv  Detail & Related papers  (2023-12-09T16:12:25Z)
• p-Laplacian Transformer [7.2541371193810384]
  $p$-Laplacian regularization, rooted in graph and image signal processing, introduces a parameter $p$ to control the regularization effect on these data. We first show that the self-attention mechanism obtains the minimal Laplacian regularization. We then propose a novel class of transformers, namely the $p$-Laplacian Transformer (p-LaT)
  arXiv  Detail & Related papers  (2023-11-06T16:25:56Z)
• Guided Patch-Grouping Wavelet Transformer with Spatial Congruence for Ultra-High Resolution Segmentation [18.50799240622156]
  Proposed Guided Patch-Grouping Wavelet Transformer (GPWFormer) $mathcalT$ takes the whole UHR image as input and harvests both local details and fine-grained long-range contextual dependencies. $mathcalC$ takes downsampled image as input for learning the category-wise deep context.
  arXiv  Detail & Related papers  (2023-07-03T02:19:48Z)
• Long-term Leap Attention, Short-term Periodic Shift for Video Classification [41.87505528859225]
  Video transformer naturally incurs a heavier computation burden than a static vision transformer. We propose the LAPS, a long-term textbftextitLeap Attention'' (LAN), short-term textbftextitPeriodic Shift'' (textitP-Shift) module for video transformers.
  arXiv  Detail & Related papers  (2022-07-12T13:30:15Z)
• Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning [77.22019100456595]
  We show a training algorithm for distributed computation workers with varying communication frequency. In this work, we obtain a tighter convergence rate of $mathcalO!!!(sigma2-2_avg!! . We also show that the heterogeneity term in rate is affected by the average delay within each worker.
  arXiv  Detail & Related papers  (2022-06-16T17:10:57Z)
• Learning Tracking Representations via Dual-Branch Fully Transformer Networks [82.21771581817937]
  We present a Siamese-like Dual-branch network based on solely Transformers for tracking. We extract a feature vector for each patch based on its matching results with others within an attention window. The method achieves better or comparable results as the best-performing methods.
  arXiv  Detail & Related papers  (2021-12-05T13:44:33Z)
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers [71.31712741938837]
  We show that sparse Transformers with only $O(n)$ connections per attention layer can approximate the same function class as the dense model with $n2$ connections. We also present experiments comparing different patterns/levels of sparsity on standard NLP tasks.
  arXiv  Detail & Related papers  (2020-06-08T18:30:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.