Fredformer: Frequency Debiased Transformer for Time Series Forecasting

• URL: http://arxiv.org/abs/2406.09009v4
• Date: Wed, 3 Jul 2024 14:24:39 GMT
• Title: Fredformer: Frequency Debiased Transformer for Time Series Forecasting
• Authors: Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, Yasushi Sakurai,
• Abstract summary: The Transformer model has shown leading performance in time series forecasting. It tends to learn low-frequency features in the data and overlook high-frequency features, showing a frequency bias. We propose Fredformer, a framework designed to mitigate frequency bias by learning features equally across different frequency bands.
• Score: 8.356290446630373
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Transformer model has shown leading performance in time series forecasting. Nevertheless, in some complex scenarios, it tends to learn low-frequency features in the data and overlook high-frequency features, showing a frequency bias. This bias prevents the model from accurately capturing important high-frequency data features. In this paper, we undertook empirical analyses to understand this bias and discovered that frequency bias results from the model disproportionately focusing on frequency features with higher energy. Based on our analysis, we formulate this bias and propose Fredformer, a Transformer-based framework designed to mitigate frequency bias by learning features equally across different frequency bands. This approach prevents the model from overlooking lower amplitude features important for accurate forecasting. Extensive experiments show the effectiveness of our proposed approach, which can outperform other baselines in different real-world time-series datasets. Furthermore, we introduce a lightweight variant of the Fredformer with an attention matrix approximation, which achieves comparable performance but with much fewer parameters and lower computation costs. The code is available at: https://github.com/chenzRG/Fredformer

Related papers

• FM-TS: Flow Matching for Time Series Generation [71.31148785577085]
  We introduce FM-TS, a rectified Flow Matching-based framework for Time Series generation. FM-TS is more efficient in terms of training and inference. We have achieved superior performance in solar forecasting and MuJoCo imputation tasks.
  arXiv  Detail & Related papers  (2024-11-12T03:03:23Z)
• Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts [103.725112190618]
  This paper introduces Moirai-MoE, using a single input/output projection layer while delegating the modeling of diverse time series patterns to the sparse mixture of experts. Extensive experiments on 39 datasets demonstrate the superiority of Moirai-MoE over existing foundation models in both in-distribution and zero-shot scenarios.
  arXiv  Detail & Related papers  (2024-10-14T13:01:11Z)
• FredNormer: Frequency Domain Normalization for Non-stationary Time Series Forecasting [18.54376910126127]
  We propose FredNormer, which observes datasets from a frequency perspective and adaptively up-weights the key frequency components. FredNormer is a plug-and-play module, which does not compromise the efficiency compared to existing normalization methods.
  arXiv  Detail & Related papers  (2024-10-02T04:05:50Z)
• Not All Frequencies Are Created Equal:Towards a Dynamic Fusion of Frequencies in Time-Series Forecasting [9.615808695919647]
  Time series forecasting methods should be flexible when applied to different scenarios. We propose Frequency Dynamic Fusion (FreDF), which individually predicts each Fourier component, and dynamically fuses the output of different frequencies.
  arXiv  Detail & Related papers  (2024-07-17T08:54:41Z)
• Deep Frequency Derivative Learning for Non-stationary Time Series Forecasting [12.989064148254936]
  We present a deep frequency derivative learning framework, DERITS, for non-stationary time series forecasting. Specifically, DERITS is built upon a novel reversible transformation, namely Frequency Derivative Transformation (FDT)
  arXiv  Detail & Related papers  (2024-06-29T17:56:59Z)
• FreDF: Learning to Forecast in Frequency Domain [56.24773675942897]
  Time series modeling is uniquely challenged by the presence of autocorrelation in both historical and label sequences. We introduce the Frequency-enhanced Direct Forecast (FreDF) which bypasses the complexity of label autocorrelation by learning to forecast in the frequency domain.
  arXiv  Detail & Related papers  (2024-02-04T08:23:41Z)
• WFTNet: Exploiting Global and Local Periodicity in Long-term Time Series Forecasting [61.64303388738395]
  We propose a Wavelet-Fourier Transform Network (WFTNet) for long-term time series forecasting. Tests on various time series datasets show WFTNet consistently outperforms other state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-09-20T13:44:18Z)
• FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecasting [21.933798421967225]
  Time series forecasting is a long-standing challenge due to the real-world information is in various scenario. We believe it's the reason that model's lacking ability of capturing frequency information which richly contains in real world datasets. We propose a novel frequency enhanced channel attention that adaptively modelling frequency interdependencies between channels based on Discrete Cosine Transform.
  arXiv  Detail & Related papers  (2022-12-02T14:40:55Z)
• Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
  We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time. This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
  arXiv  Detail & Related papers  (2022-11-26T01:56:05Z)
• Simpler is better: spectral regularization and up-sampling techniques for variational autoencoders [1.2234742322758418]
  characterization of the spectral behavior of generative models based on neural networks remains an open issue. Recent research has focused heavily on generative adversarial networks and the high-frequency discrepancies between real and generated images. We propose a simple 2D Fourier transform-based spectral regularization loss for the Variational Autoencoders (VAEs)
  arXiv  Detail & Related papers  (2022-01-19T11:49:57Z)
• Real Time Speech Enhancement in the Waveform Domain [99.02180506016721]
  We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is capable of removing various kinds of background noise including stationary and non-stationary noises.
  arXiv  Detail & Related papers  (2020-06-23T09:19:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.