Enabling Time-series Foundation Model for Building Energy Forecasting via Contrastive Curriculum Learning

• URL: http://arxiv.org/abs/2412.17285v1
• Date: Mon, 23 Dec 2024 05:07:06 GMT
• Title: Enabling Time-series Foundation Model for Building Energy Forecasting via Contrastive Curriculum Learning
• Authors: Rui Liang, Yang Deng, Donghua Xie, Fang He, Dan Wang,
• Abstract summary: We study the adaptation of foundation models (FMs) to building energy forecasting tasks. We propose a new textitcontrastive curriculum learning-based training method. Experiments show that our method can improve the zero/few-shot performance by 14.6% compared to the existing FMs.
• Score: 12.19823790689484
• License:
• Abstract: Advances in time-series forecasting are driving a shift from conventional machine learning models to foundation models (FMs) that are trained with generalized knowledge. However, existing FMs still perform poorly in the energy fields, such as building energy forecasting (BEF). This paper studies the adaptation of FM to BEF tasks. We demonstrate the shortcomings of fine-tuning FM straightforwardly from both the perspectives of FM and the data. To overcome these limitations, we propose a new \textit{contrastive curriculum learning}-based training method. Our method optimizes the ordering of training data in the context of TSFM adaptation. Experiments show that our method can improve the zero/few-shot performance by 14.6\% compared to the existing FMs. Our code and new TSFM will be available at <Anonymous Github Repo>.

Related papers

• Lightweight Online Adaption for Time Series Foundation Model Forecasts [0.32622301272834525]
  AdapTS is a lightweight mechanism for the online adaption of FM forecasts in response to online feedback. We evaluate the performance of AdapTS in conjunction with several recent FMs across a suite of standard time series datasets.
  arXiv  Detail & Related papers  (2025-02-18T15:01:02Z)
• A Framework for Double-Blind Federated Adaptation of Foundation Models [4.910367774892893]
  We propose a framework for double-blind federated adaptation of FMs using fully homomorphic encryption (FHE) The proposed framework first decomposes the FM into a sequence of FHE-friendly blocks through knowledge distillation. The resulting FHE-friendly model is adapted for the downstream task via low-rank parallel adapters.
  arXiv  Detail & Related papers  (2025-02-03T12:00:11Z)
• Synergizing Foundation Models and Federated Learning: A Survey [23.416321895575507]
  This paper discusses the potentials and challenges of synergizing Federated Learning (FL) and Foundation Models (FM) FL is a collaborative learning paradigm that breaks the barrier of data availability from different participants. It provides a promising solution to customize and adapt FMs to a wide range of domain-specific tasks using distributed datasets whilst preserving privacy.
  arXiv  Detail & Related papers  (2024-06-18T17:58:09Z)
• FedPFT: Federated Proxy Fine-Tuning of Foundation Models [55.58899993272904]
  Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges as a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. We propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules.
  arXiv  Detail & Related papers  (2024-04-17T16:30:06Z)
• Optimal Flow Matching: Learning Straight Trajectories in Just One Step [89.37027530300617]
  We develop and theoretically justify the novel textbf Optimal Flow Matching (OFM) approach. It allows recovering the straight OT displacement for the quadratic transport in just one FM step. The main idea of our approach is the employment of vector field for FM which are parameterized by convex functions.
  arXiv  Detail & Related papers  (2024-03-19T19:44:54Z)
• A Survey on Efficient Federated Learning Methods for Foundation Model Training [62.473245910234304]
  Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
  arXiv  Detail & Related papers  (2024-01-09T10:22:23Z)
• Learning from models beyond fine-tuning [78.20895343699658]
  Learn From Model (LFM) focuses on the research, modification, and design of foundation models (FM) based on the model interface. The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing. This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM.
  arXiv  Detail & Related papers  (2023-10-12T10:20:36Z)
• VideoGLUE: Video General Understanding Evaluation of Foundation Models [89.07145427268948]
  We evaluate video understanding capabilities of foundation models (FMs) using a carefully designed experiment protocol. We jointly profile FMs' hallmark and efficacy efficiency when adapting to general video understanding tasks.
  arXiv  Detail & Related papers  (2023-07-06T17:47:52Z)
• Improving and generalizing flow-based generative models with minibatch optimal transport [90.01613198337833]
  We introduce the generalized conditional flow matching (CFM) technique for continuous normalizing flows (CNFs) CFM features a stable regression objective like that used to train the flow in diffusion models but enjoys the efficient inference of deterministic flow models. A variant of our objective is optimal transport CFM (OT-CFM), which creates simpler flows that are more stable to train and lead to faster inference.
  arXiv  Detail & Related papers  (2023-02-01T14:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.