Low-rank Adaptation for Spatio-Temporal Forecasting

• URL: http://arxiv.org/abs/2404.07919v1
• Date: Thu, 11 Apr 2024 17:04:55 GMT
• Title: Low-rank Adaptation for Spatio-Temporal Forecasting
• Authors: Weilin Ruan, Wei Chen, Xilin Dang, Jianxiang Zhou, Weichuang Li, Xu Liu, Yuxuan Liang,
• Abstract summary: We present a novel low-rank adaptation framework as an off-the-shelf plugin for existing spatialtemporal prediction models, STLo-RA. Our approach increases parameters and training time of the original models by less than 4%, still achieving consistent and sustained performance enhancement.
• Score: 13.595533573828734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatio-temporal forecasting is crucial in real-world dynamic systems, predicting future changes using historical data from diverse locations. Existing methods often prioritize the development of intricate neural networks to capture the complex dependencies of the data, yet their accuracy fails to show sustained improvement. Besides, these methods also overlook node heterogeneity, hindering customized prediction modules from handling diverse regional nodes effectively. In this paper, our goal is not to propose a new model but to present a novel low-rank adaptation framework as an off-the-shelf plugin for existing spatial-temporal prediction models, termed ST-LoRA, which alleviates the aforementioned problems through node-level adjustments. Specifically, we first tailor a node adaptive low-rank layer comprising multiple trainable low-rank matrices. Additionally, we devise a multi-layer residual fusion stacking module, injecting the low-rank adapters into predictor modules of various models. Across six real-world traffic datasets and six different types of spatio-temporal prediction models, our approach minimally increases the parameters and training time of the original models by less than 4%, still achieving consistent and sustained performance enhancement.

Related papers

• Robust Traffic Forecasting against Spatial Shift over Years [11.208740750755025]
  We investigate state-temporal-the-art models using newly proposed traffic OOD benchmarks. We find that these models experience significant decline in performance. We propose a novel of Mixture Experts framework, which learns a set of graph generators during training and combines them to generate new graphs. Our method is both parsimonious and efficacious, and can be seamlessly integrated into anytemporal model.
  arXiv  Detail & Related papers  (2024-10-01T03:49:29Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation [56.79064699832383]
  We establish a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm in which the edge models only need to perform forward propagation. In our CEMA, to reduce the communication burden, we devise two criteria to exclude unnecessary samples from uploading to the cloud.
  arXiv  Detail & Related papers  (2024-02-27T08:47:19Z)
• RPMixer: Shaking Up Time Series Forecasting with Random Projections for Large Spatial-Temporal Data [33.0546525587517]
  We propose a all-Multi-Layer Perceptron (all-MLP) time series forecasting architecture called RPMixer. Our method capitalizes on the ensemble-like behavior of deep neural networks, where each individual block behaves like a base learner in an ensemble model.
  arXiv  Detail & Related papers  (2024-02-16T07:28:59Z)
• Learning Robust Precipitation Forecaster by Temporal Frame Interpolation [65.5045412005064]
  We develop a robust precipitation forecasting model that demonstrates resilience against spatial-temporal discrepancies. Our approach has led to significant improvements in forecasting precision, culminating in our model securing textit1st place in the transfer learning leaderboard of the textitWeather4cast'23 competition.
  arXiv  Detail & Related papers  (2023-11-30T08:22:08Z)
• Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations [15.797295258800638]
  We introduce a novel modeling approach for time series imputation and forecasting, tailored to address the challenges often encountered in real-world data. Our method relies on a continuous-time-dependent model of the series' evolution dynamics. A modulation mechanism, driven by a meta-learning algorithm, allows adaptation to unseen samples and extrapolation beyond observed time-windows.
  arXiv  Detail & Related papers  (2023-06-09T13:20:04Z)
• Dynamic Graph Neural Network with Adaptive Edge Attributes for Air Quality Predictions [12.336689498639366]
  We propose a novel Dynamic Graph Neural Network with Adaptive Edge Attributes (DGN-AEA) on the message passing network. Unlike prior information to establish edges, our method can obtain adaptive edge information through end-to-end training without any prior information.
  arXiv  Detail & Related papers  (2023-02-20T13:45:55Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder [59.69303945834122]
  We present a Smoothness-Inducing Sequential Variational Auto-Encoder (SISVAE) model for robust estimation and anomaly detection of time series. Our model parameterizes mean and variance for each time-stamp with flexible neural networks. We show the effectiveness of our model on both synthetic datasets and public real-world benchmarks.
  arXiv  Detail & Related papers  (2021-02-02T06:15:15Z)
• Learning Consistent Deep Generative Models from Sparse Data via Prediction Constraints [16.48824312904122]
  We develop a new framework for learning variational autoencoders and other deep generative models. We show that these two contributions -- prediction constraints and consistency constraints -- lead to promising image classification performance.
  arXiv  Detail & Related papers  (2020-12-12T04:18:50Z)
• Pre-Trained Models for Heterogeneous Information Networks [57.78194356302626]
  We propose a self-supervised pre-training and fine-tuning framework, PF-HIN, to capture the features of a heterogeneous information network. PF-HIN consistently and significantly outperforms state-of-the-art alternatives on each of these tasks, on four datasets.
  arXiv  Detail & Related papers  (2020-07-07T03:36:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.