A Supervised Contrastive Learning Pretrain-Finetune Approach for Time
Series
- URL: http://arxiv.org/abs/2311.12290v1
- Date: Tue, 21 Nov 2023 02:06:52 GMT
- Title: A Supervised Contrastive Learning Pretrain-Finetune Approach for Time
Series
- Authors: Trang H. Tran, Lam M. Nguyen, Kyongmin Yeo, Nam Nguyen, Roman Vaculin
- Abstract summary: We introduce a novel pretraining procedure that leverages supervised contrastive learning to distinguish features within each pretraining dataset.
We then propose a fine-tuning procedure designed to enhance the accurate prediction of the target data by aligning it more closely with the learned dynamics of the pretraining datasets.
- Score: 15.218841180577135
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Foundation models have recently gained attention within the field of machine
learning thanks to its efficiency in broad data processing. While researchers
had attempted to extend this success to time series models, the main challenge
is effectively extracting representations and transferring knowledge from
pretraining datasets to the target finetuning dataset. To tackle this issue, we
introduce a novel pretraining procedure that leverages supervised contrastive
learning to distinguish features within each pretraining dataset. This
pretraining phase enables a probabilistic similarity metric, which assesses the
likelihood of a univariate sample being closely related to one of the
pretraining datasets. Subsequently, using this similarity metric as a guide, we
propose a fine-tuning procedure designed to enhance the accurate prediction of
the target data by aligning it more closely with the learned dynamics of the
pretraining datasets. Our experiments have shown promising results which
demonstrate the efficacy of our approach.
Related papers
- Prediction of rare events in the operation of household equipment using
co-evolving time series [1.1249583407496218]
Our approach involves a weighted autologistic regression model, where we leverage the temporal behavior of the data to enhance predictive capabilities.
Evaluation on synthetic and real-world datasets confirms that our approach outperform state-of-the-art of predicting home equipment failure methods.
arXiv Detail & Related papers (2023-12-15T00:21:00Z) - Examining the Effect of Pre-training on Time Series Classification [21.38211396933795]
This study investigates the impact of pre-training followed by fine-tuning on the fine-tuning process.
We conducted a thorough examination of 150 classification datasets.
We find that pre-training can only help improve the optimization process for models that fit the data poorly.
Adding more pre-training data does not improve generalization, but it can strengthen the advantage of pre-training on the original data volume.
arXiv Detail & Related papers (2023-09-11T06:26:57Z) - On the Trade-off of Intra-/Inter-class Diversity for Supervised
Pre-training [72.8087629914444]
We study the impact of the trade-off between the intra-class diversity (the number of samples per class) and the inter-class diversity (the number of classes) of a supervised pre-training dataset.
With the size of the pre-training dataset fixed, the best downstream performance comes with a balance on the intra-/inter-class diversity.
arXiv Detail & Related papers (2023-05-20T16:23:50Z) - Towards Out-of-Distribution Sequential Event Prediction: A Causal
Treatment [72.50906475214457]
The goal of sequential event prediction is to estimate the next event based on a sequence of historical events.
In practice, the next-event prediction models are trained with sequential data collected at one time.
We propose a framework with hierarchical branching structures for learning context-specific representations.
arXiv Detail & Related papers (2022-10-24T07:54:13Z) - Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
We propose self-distillation as a regularization for a further pre-training stage.
We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
arXiv Detail & Related papers (2022-09-30T02:25:12Z) - Improved Fine-tuning by Leveraging Pre-training Data: Theory and
Practice [52.11183787786718]
Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications.
Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy.
We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
arXiv Detail & Related papers (2021-11-24T06:18:32Z) - Learning PAC-Bayes Priors for Probabilistic Neural Networks [32.01506699213665]
Recent works have investigated deep learning models trained by optimising PAC-Bayes bounds, with priors that are learnt on subsets of the data.
We ask what is the optimal amount of data which should be allocated for building the prior and show that the optimum may be dataset dependent.
arXiv Detail & Related papers (2021-09-21T16:27:42Z) - Video Prediction via Example Guidance [156.08546987158616]
In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics.
In this work, we propose a simple yet effective framework that can efficiently predict plausible future states.
arXiv Detail & Related papers (2020-07-03T14:57:24Z) - Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training.
We experimentally verify that the new dataset can significantly improve the ability of the learned FER model.
To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
arXiv Detail & Related papers (2020-05-18T09:36:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.