Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?

• URL: http://arxiv.org/abs/2201.04449v1
• Date: Wed, 12 Jan 2022 12:55:21 GMT
• Title: Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?
• Authors: Erik Otovi\'c, Marko Njirjak, Dario Jozinovi\'c, Goran Mau\v{s}a, Alberto Michelini, Ivan \v{S}tajduhar
• Abstract summary: This study aims to assess how transferable are the features between different domains of time series data. The effects of transfer learning are observed in terms of predictive performance of the models and their convergence rate during training.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In practice, it is very demanding and sometimes impossible to collect datasets of tagged data large enough to successfully train a machine learning model, and one possible solution to this problem is transfer learning. This study aims to assess how transferable are the features between different domains of time series data and under which conditions. The effects of transfer learning are observed in terms of predictive performance of the models and their convergence rate during training. In our experiment, we use reduced data sets of 1,500 and 9,000 data instances to mimic real world conditions. Using the same scaled-down datasets, we trained two sets of machine learning models: those that were trained with transfer learning and those that were trained from scratch. Four machine learning models were used for the experiment. Transfer of knowledge was performed within the same domain of application (seismology), as well as between mutually different domains of application (seismology, speech, medicine, finance). We observe the predictive performance of the models and the convergence rate during the training. In order to confirm the validity of the obtained results, we repeated the experiments seven times and applied statistical tests to confirm the significance of the results. The general conclusion of our study is that transfer learning is very likely to either increase or not negatively affect the predictive performance of the model or its convergence rate. The collected data is analysed in more details to determine which source and target domains are compatible for transfer of knowledge. We also analyse the effect of target dataset size and the selection of model and its hyperparameters on the effects of transfer learning.

Related papers

• Diffusion-based Neural Network Weights Generation [85.6725307453325]
  We propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Transferability of Representations Learned using Supervised Contrastive Learning Trained on a Multi-Domain Dataset [5.389242063238166]
  Contrastive learning has shown to learn better quality representations than models trained using cross-entropy loss. This paper explores the transferability of representations learned using contrastive learning when trained on a multi-domain dataset.
  arXiv  Detail & Related papers  (2023-09-27T08:34:36Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Transfer Learning between Motor Imagery Datasets using Deep Learning -- Validation of Framework and Comparison of Datasets [0.0]
  We present a simple deep learning-based framework commonly used in computer vision. We demonstrate its effectiveness for cross-dataset transfer learning in mental imagery decoding tasks.
  arXiv  Detail & Related papers  (2023-09-04T20:58:57Z)
• An Exploration of Data Efficiency in Intra-Dataset Task Transfer for Dialog Understanding [65.75873687351553]
  This study explores the effects of varying quantities of target task training data on sequential transfer learning in the dialog domain. Unintuitively, our data shows that often target task training data size has minimal effect on how sequential transfer learning performs compared to the same model without transfer learning.
  arXiv  Detail & Related papers  (2022-10-21T04:36:46Z)
• How Well Do Sparse Imagenet Models Transfer? [75.98123173154605]
  Transfer learning is a classic paradigm by which models pretrained on large "upstream" datasets are adapted to yield good results on "downstream" datasets. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset. We show that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities.
  arXiv  Detail & Related papers  (2021-11-26T11:58:51Z)
• A Systematic Evaluation of Domain Adaptation in Facial Expression Recognition [0.0]
  This paper provides a systematic evaluation of domain adaptation in facial expression recognition. We use state-of-the-art transfer learning techniques and six commonly-used facial expression datasets. We find sobering results that the accuracy of transfer learning is not high, and varies idiosyncratically with the target dataset.
  arXiv  Detail & Related papers  (2021-06-29T14:41:19Z)
• Deep Transfer Learning for Infectious Disease Case Detection Using Electronic Medical Records [0.0]
  During an infectious disease pandemic, it is critical to share electronic medical records or models (learned from these records) across regions. Applying one region's data/model to another region often have distribution shift issues that violate the assumptions of traditional machine learning techniques. To explore the potential of deep transfer learning algorithms, we applied two data-based algorithms and model-based transfer learning algorithms to infectious disease detection tasks.
  arXiv  Detail & Related papers  (2021-03-08T01:53:29Z)
• What is being transferred in transfer learning? [51.6991244438545]
  We show that when training from pre-trained weights, the model stays in the same basin in the loss landscape. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.
  arXiv  Detail & Related papers  (2020-08-26T17:23:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.