How Good Can Linear Models Be for Time-Series Forecasting?

• URL: http://arxiv.org/abs/2606.27282v2
• Date: Mon, 29 Jun 2026 17:59:36 GMT
• Title: How Good Can Linear Models Be for Time-Series Forecasting?
• Abstract summary: We use Ridge regression as the testbed, since it has a closed-form solution and interpretable weights.<n>We search over context length, local normalization, regularization, and augmentation on eight standard benchmarks.<n>The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, CNN baselines on six of eight benchmarks.
• Score: 1.558068541168739
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Time-series forecasting research has been moving steadily toward larger architectures, from specialized transformers to general-purpose foundation models, on the assumption that capacity is what unlocks accuracy. We take the opposite position: most of the gap can be closed at far lower cost by tuning preprocessing rather than scaling models. We use Ridge regression as the testbed, since it has a closed-form solution and interpretable weights, which let the optimal hyperparameters be read off the search directly. We search over context length, local normalization, regularization, and augmentation on eight standard benchmarks and find three patterns. (1) Optimal lookback is strongly series-specific and often non-monotonic in forecast horizon, with fitted power-law exponents ranging from $+0.46$ on ETTm2 to $-0.19$ on Exchange and Traffic, challenging the convention that longer horizons need longer history. (2) Normalizing over a learned trailing fraction of the context, rather than its entirety, is almost universally preferred. (3) Series within the same dataset often disagree on hyperparameters; the optimal degree of cross-series sharing varies from fully shared to fully per-series. The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, MLP, and CNN baselines on six of eight benchmarks. The optimized hyperparameters also serve as a diagnostic on the data itself, revealing structures that larger models absorb silently into their learned parameters. We provide an accompanying interactive online demonstration and the code at https://sakanaai.github.io/SearchCast/.

Related papers

• Revisiting Chebyshev Polynomial and Anisotropic RBF Models for Tabular Regression [0.7734726150561086]
  Smooth-basis models such as Chebyshev regressors and radial basis function (RBF) networks are well established in numerical analysis.<n>We ask whether they can compete, benchmarking models across 55 regression datasets organised by application domain.
  arXiv  Detail & Related papers  (2026-02-25T21:34:52Z)
• OverFill: Two-Stage Models for Efficient Language Model Decoding [68.68408155020568]
  Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs.<n>We propose OverFill, which decouples prefill and decode stages to optimize accuracy-efficiency tradeoffs.<n>Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average.
  arXiv  Detail & Related papers  (2025-08-11T20:07:34Z)
• T3Time: Tri-Modal Time Series Forecasting via Adaptive Multi-Head Alignment and Residual Fusion [0.4915744683251151]
  T3Time is a novel trimodal framework consisting of time, spectral, and prompt branches.<n>It learns prioritization between temporal and spectral features based on the prediction horizon.<n>Our model consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-06T09:31:44Z)
• Intention-Conditioned Flow Occupancy Models [80.42634994902858]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain [24.24834151329251]
  OLinear is a $mathbfo$rthogonally transformed domain that operates in a $mathbfo$rthogonally transformed domain.<n>We introduce a customized linear layer, $mathbfNormLin$, which employs a normalized weight matrix to capture multivariate dependencies.<n>Experiments on 24 benchmarks and 140 forecasting tasks demonstrate that OLinear consistently achieves state-of-the-art performance with high efficiency.
  arXiv  Detail & Related papers  (2025-05-12T10:39:37Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  We conduct an unprecedented empirical investigation training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law.<n>Our estimated optima deviates from the global best performance found via exhaustive search by merely 0.094% on the test set.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)
• SWIFT: Mapping Sub-series with Wavelet Decomposition Improves Time Series Forecasting [2.6764607949560593]
  $textitSWIFT$ is a lightweight model that is powerful, but also efficient in deployment and inference for Long-term Time Series Forecasting.<n>We conduct comprehensive experiments, and the results show that $textitSWIFT$ achieves state-of-the-art (SOTA) performance on multiple datasets.
  arXiv  Detail & Related papers  (2025-01-27T16:26:07Z)
• tsGT: Stochastic Time Series Modeling With Transformer [0.12905935507312413]
  We introduce tsGT, a time series model built on a general-purpose transformer architecture. We show that tsGT outperforms the state-of-the-art models on MAD and RMSE, and surpasses its peers on QL and CRPS, on four commonly used datasets.
  arXiv  Detail & Related papers  (2024-03-08T22:59:41Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost [53.746169882193456]
  Recent works have proposed various sparse attention modules to overcome the quadratic cost of self-attention. We propose a model that resolves both problems by endowing each attention head with a mixed-membership Block Model. Our model outperforms previous efficient variants as well as the original Transformer with full attention.
  arXiv  Detail & Related papers  (2022-10-27T15:30:52Z)
• High-dimensional regression with potential prior information on variable importance [0.0]
  We propose a simple scheme involving fitting a sequence of models indicated by the ordering. We show that the computational cost for fitting all models when ridge regression is used is no more than for a single fit of ridge regression. We describe a strategy for Lasso regression that makes use of previous fits to greatly speed up fitting the entire sequence of models.
  arXiv  Detail & Related papers  (2021-09-23T10:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.