FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain

• URL: http://arxiv.org/abs/2412.01654v2
• Date: Tue, 03 Dec 2024 04:40:13 GMT
• Title: FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain
• Authors: Zhengnan Li, Haoxuan Li, Hao Wang, Jun Fang, Duoyin Li Yunxiao Qin,
• Abstract summary: Time series forecasting (TSF) plays a crucial role in various domains, including web data analysis, energy consumption prediction, and weather forecasting. While Multi-Layer Perceptrons (MLPs) are lightweight and effective for capturing temporal dependencies, they are prone to overfitting when used to model inter-channel dependencies. We introduce a novel Simplex-MLP layer, where the weights are constrained within a standard simplex. This strategy encourages the model to learn simpler patterns and thereby reducing overfitting to extreme values.
• Score: 16.693117400535833
• License:
• Abstract: Time series forecasting (TSF) plays a crucial role in various domains, including web data analysis, energy consumption prediction, and weather forecasting. While Multi-Layer Perceptrons (MLPs) are lightweight and effective for capturing temporal dependencies, they are prone to overfitting when used to model inter-channel dependencies. In this paper, we investigate the overfitting problem in channel-wise MLPs using Rademacher complexity theory, revealing that extreme values in time series data exacerbate this issue. To mitigate this issue, we introduce a novel Simplex-MLP layer, where the weights are constrained within a standard simplex. This strategy encourages the model to learn simpler patterns and thereby reducing overfitting to extreme values. Based on the Simplex-MLP layer, we propose a novel \textbf{F}requency \textbf{S}implex \textbf{MLP} (FSMLP) framework for time series forecasting, comprising of two kinds of modules: \textbf{S}implex \textbf{C}hannel-\textbf{W}ise MLP (SCWM) and \textbf{F}requency \textbf{T}emporal \textbf{M}LP (FTM). The SCWM effectively leverages the Simplex-MLP to capture inter-channel dependencies, while the FTM is a simple yet efficient temporal MLP designed to extract temporal information from the data. Our theoretical analysis shows that the upper bound of the Rademacher Complexity for Simplex-MLP is lower than that for standard MLPs. Moreover, we validate our proposed method on seven benchmark datasets, demonstrating significant improvements in forecasting accuracy and efficiency, while also showcasing superior scalability. Additionally, we demonstrate that Simplex-MLP can improve other methods that use channel-wise MLP to achieve less overfitting and improved performance. Code are available \href{https://github.com/FMLYD/FSMLP}{\textcolor{red}{here}}.

Related papers

• CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning [59.88924847995279]
  We propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF. To reduce the distribution discrepancy, we develop the cross-modal match module. CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks.
  arXiv  Detail & Related papers  (2024-03-12T04:04:38Z)
• Is Vanilla MLP in Neural Radiance Field Enough for Few-shot View Synthesis? [19.34823662319042]
  NeRF has achieved superior performance for novel view synthesis by modeling the scene with a Multi-Layer Perception (MLP) and a volume rendering procedure. When fewer known views are given, the model is prone to overfitting the given views.
  arXiv  Detail & Related papers  (2024-03-10T04:27:06Z)
• Frequency-domain MLPs are More Effective Learners in Time Series Forecasting [67.60443290781988]
  Time series forecasting has played the key role in different industrial domains, including finance, traffic, energy, and healthcare. Most-based forecasting methods suffer from the point-wise mappings and information bottleneck. We propose FreTS, a simple yet effective architecture built upon Frequency-domains for Time Series forecasting.
  arXiv  Detail & Related papers  (2023-11-10T17:05:13Z)
• Strip-MLP: Efficient Token Interaction for Vision MLP [31.02197585697145]
  We introduce textbfStrip-MLP to enrich the token interaction power in three ways. Strip-MLP significantly improves the performance of spatial-based models on small datasets. Models achieve higher average Top-1 accuracy than existing datasets by +2.44% on Caltech-101 and +2.16% on CIFAR-100.
  arXiv  Detail & Related papers  (2023-07-21T09:40:42Z)
• MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models [33.86069537521178]
  Fine-tuning a pre-trained language model (PLM) emerges as the predominant strategy in many natural language processing applications. General approaches (e.g. quantization and distillation) have been widely studied to reduce the compute/memory of PLM fine-tuning. We propose one-shot compression techniques specifically designed for fine-tuning.
  arXiv  Detail & Related papers  (2023-07-18T03:12:51Z)
• UNeXt: MLP-based Rapid Medical Image Segmentation Network [80.16644725886968]
  UNet and its latest extensions like TransUNet have been the leading medical image segmentation methods in recent years. We propose UNeXt which is a Convolutional multilayer perceptron based network for image segmentation. We show that we reduce the number of parameters by 72x, decrease the computational complexity by 68x, and improve the inference speed by 10x while also obtaining better segmentation performance.
  arXiv  Detail & Related papers  (2022-03-09T18:58:22Z)
• Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework [55.40001810884942]
  We introduce a pure residual network, called PointMLP, which integrates no sophisticated local geometrical extractors but still performs very competitively. On the real-world ScanObjectNN dataset, our method even surpasses the prior best method by 3.3% accuracy. Compared to most recent CurveNet, PointMLP trains 2x faster, tests 7x faster, and is more accurate on ModelNet40 benchmark.
  arXiv  Detail & Related papers  (2022-02-15T01:39:07Z)
• RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality [113.1414517605892]
  We propose a methodology, Locality Injection, to incorporate local priors into an FC layer. RepMLPNet is the first that seamlessly transfer to Cityscapes semantic segmentation.
  arXiv  Detail & Related papers  (2021-12-21T10:28:17Z)
• Sparse-MLP: A Fully-MLP Architecture with Conditional Computation [7.901786481399378]
  Mixture-of-Experts (MoE) with sparse conditional computation has been proved an effective architecture for scaling attention-based models to more parameters with comparable computation cost. We propose Sparse-MLP, scaling the recent-Mixer model with MoE, to achieve a more-efficient architecture.
  arXiv  Detail & Related papers  (2021-09-05T06:43:08Z)
• AS-MLP: An Axial Shifted MLP Architecture for Vision [50.11765148947432]
  An Axial Shifted architecture (AS-MLP) is proposed in this paper. By axially shifting channels of the feature map, AS-MLP is able to obtain the information flow from different directions. With the proposed AS-MLP architecture, our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset.
  arXiv  Detail & Related papers  (2021-07-18T08:56:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.