GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts

• URL: http://arxiv.org/abs/2508.17515v1
• Date: Sun, 24 Aug 2025 20:39:50 GMT
• Title: GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts
• Authors: Kyrylo Yemets, Mykola Lukashchuk, Ivan Izonin,
• Abstract summary: We propose a model architecture that simplifies the training process for univariate time series forecasting.<n>Our approach combines sparse MoE computation with a novel attention-inspired gating mechanism that replaces the traditional one-layer softmax router.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accurate univariate forecasting remains a pressing need in real-world systems, such as energy markets, hydrology, retail demand, and IoT monitoring, where signals are often intermittent and horizons span both short- and long-term. While transformers and Mixture-of-Experts (MoE) architectures are increasingly favored for time-series forecasting, a key gap persists: MoE models typically require complicated training with both the main forecasting loss and auxiliary load-balancing losses, along with careful routing/temperature tuning, which hinders practical adoption. In this paper, we propose a model architecture that simplifies the training process for univariate time series forecasting and effectively addresses both long- and short-term horizons, including intermittent patterns. Our approach combines sparse MoE computation with a novel attention-inspired gating mechanism that replaces the traditional one-layer softmax router. Through extensive empirical evaluation, we demonstrate that our gating design naturally promotes balanced expert utilization and achieves superior predictive accuracy without requiring the auxiliary load-balancing losses typically used in classical MoE implementations. The model achieves better performance while utilizing only a fraction of the parameters required by state-of-the-art transformer models, such as PatchTST. Furthermore, experiments across diverse datasets confirm that our MoE architecture with the proposed gating mechanism is more computationally efficient than LSTM for both long- and short-term forecasting, enabling cost-effective inference. These results highlight the potential of our approach for practical time-series forecasting applications where both accuracy and computational efficiency are critical.

Related papers

• A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs [64.8510381475827]
  Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently.<n>SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized.<n>We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set.
  arXiv  Detail & Related papers  (2026-02-23T15:11:16Z)
• MEMTS: Internalizing Domain Knowledge via Parameterized Memory for Retrieval-Free Domain Adaptation of Time Series Foundation Models [51.506429027626005]
  Memory for Time Series (MEMTS) is a lightweight and plug-and-play method for retrieval-free domain adaptation in time series forecasting.<n>Key component of MEMTS is a Knowledge Persistence Module (KPM), which internalizes domain-specific temporal dynamics.<n>This paradigm shift enables MEMTS to achieve accurate domain adaptation with constant-time inference and near-zero latency.
  arXiv  Detail & Related papers  (2026-02-14T14:00:06Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• Distilling Time Series Foundation Models for Efficient Forecasting [28.730703685779186]
  We present DistilTS, the first distillation framework specifically designed for Time Series foundation models (TSFMs)<n>DistilTS addresses two key challenges: (1) task difficulty discrepancy, specific to forecasting, where uniform weighting makes optimization dominated by easier short-term horizons, while long-term horizons receive weaker supervision; and (2) architecture discrepancy, a general challenge in distillation, for which we design an alignment mechanism in the time series forecasting.<n> Experiments on multiple benchmarks demonstrate that DistilTS achieves forecasting performance comparable to full-sized TSFMs, while reducing parameters by up to 1/150 and accelerating inference by up to 6000x.
  arXiv  Detail & Related papers  (2026-01-19T07:32:00Z)
• Temporal convolutional and fusional transformer model with Bi-LSTM encoder-decoder for multi-time-window remaining useful life prediction [0.0]
  We propose a novel framework that integrates Temporal Convolutional Networks (TCNs) for localized temporal feature extraction.<n>This architecture effectively bridges short- and long-term dependencies while emphasizing salient temporal patterns.<n>The proposed model reduces the average RMSE by up to 5.5%, underscoring its improved predictive accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-11-06T11:29:57Z)
• Neural Architecture Search for global multi-step Forecasting of Energy Production Time Series [4.605677844197738]
  We design a neural architecture search (NAS)-based framework for the automated discovery of time series models.<n>In particular, we introduce a search space consisting only of efficient components, which can capture distinctive patterns of energy time series.<n>Results show that NAS outperforms state-of-the-art techniques, such as Transformers, in terms of both efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-10-27T15:56:37Z)
• Advanced Hybrid Transformer LSTM Technique with Attention and TS Mixer for Drilling Rate of Penetration Prediction [1.2432046687586285]
  Rate of Penetration (ROP) is crucial for optimizing drilling operations.<n>Traditional empirical, physics-based, and basic machine learning models often fail to capture intricate temporal and contextual relationships.<n>We propose a novel hybrid deep learning architecture integrating Long Short-Term Memory (LSTM) networks, Transformer encoders, Time-Series Mixer (TS-Mixer) blocks.
  arXiv  Detail & Related papers  (2025-08-07T09:45:56Z)
• Does Scaling Law Apply in Time Series Forecasting? [2.127584662240465]
  We propose Alinear, an ultra-lightweight forecasting model that achieves competitive performance using only k-level parameters.<n>Experiments on seven benchmark datasets demonstrate that Alinear consistently outperforms large-scale models.<n>This work challenges the prevailing belief that larger models are inherently better and suggests a paradigm shift toward more efficient time series modeling.
  arXiv  Detail & Related papers  (2025-05-15T11:04:39Z)
• Transformer-Based Bearing Fault Detection using Temporal Decomposition Attention Mechanism [0.40964539027092917]
  Bearing fault detection is a critical task in predictive maintenance, where accurate and timely fault identification can prevent costly downtime and equipment damage.<n>Traditional attention mechanisms in Transformer neural networks often struggle to capture the complex temporal patterns in bearing vibration data, leading to suboptimal performance.<n>We propose a novel attention mechanism, Temporal Decomposition Attention (TDA), which combines temporal bias encoding with seasonal-trend decomposition to capture both long-term dependencies and periodic fluctuations in time series data.
  arXiv  Detail & Related papers  (2024-12-15T16:51:31Z)
• Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models [68.23649978697027]
  Forecast-PEFT is a fine-tuning strategy that freezes the majority of the model's parameters, focusing adjustments on newly introduced prompts and adapters. Our experiments show that Forecast-PEFT outperforms traditional full fine-tuning methods in motion prediction tasks. Forecast-FT further improves prediction performance, evidencing up to a 9.6% enhancement over conventional baseline methods.
  arXiv  Detail & Related papers  (2024-07-28T19:18:59Z)
• Are Self-Attentions Effective for Time Series Forecasting? [4.990206466948269]
  Time series forecasting is crucial for applications across multiple domains and various scenarios.<n>Recent findings have indicated that simpler linear models might outperform complex Transformer-based approaches.<n>We introduce a new architecture, Cross-Attention-only Time Series transformer (CATS)<n>Our model achieves superior performance with the lowest mean squared error and uses fewer parameters compared to existing models.
  arXiv  Detail & Related papers  (2024-05-27T06:49:39Z)
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts [49.01990048827639]
  We introduce SEER-MoE, a framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.
  arXiv  Detail & Related papers  (2024-04-07T22:13:43Z)
• Parsimony or Capability? Decomposition Delivers Both in Long-term Time Series Forecasting [46.63798583414426]
  Long-term time series forecasting (LTSF) represents a critical frontier in time series analysis. Our study demonstrates, through both analytical and empirical evidence, that decomposition is key to containing excessive model inflation. Remarkably, by tailoring decomposition to the intrinsic dynamics of time series data, our proposed model outperforms existing benchmarks.
  arXiv  Detail & Related papers  (2024-01-22T13:15:40Z)
• Perceiver-based CDF Modeling for Time Series Forecasting [25.26713741799865]
  We propose a new architecture, called perceiver-CDF, for modeling cumulative distribution functions (CDF) of time series data. Our approach combines the perceiver architecture with a copula-based attention mechanism tailored for multimodal time series prediction. Experiments on the unimodal and multimodal benchmarks consistently demonstrate a 20% improvement over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-10-03T01:13:17Z)
• Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution [57.71199089609161]
  Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning. Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism. We propose an efficient Transformerbased model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects.
  arXiv  Detail & Related papers  (2023-01-05T13:59:29Z)
• Transformer Hawkes Process [79.16290557505211]
  We propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies. THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.
  arXiv  Detail & Related papers  (2020-02-21T13:48:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.