Transformer-Based Behavioral Representation Learning Enables Transfer Learning for Mobile Sensing in Small Datasets

• URL: http://arxiv.org/abs/2107.06097v1
• Date: Fri, 9 Jul 2021 22:26:50 GMT
• Title: Transformer-Based Behavioral Representation Learning Enables Transfer Learning for Mobile Sensing in Small Datasets
• Authors: Mike A. Merrill and Tim Althoff
• Abstract summary: We provide a neural architecture framework for mobile sensing data that can learn generalizable feature representations from time series. This architecture combines benefits from CNN and Trans-former architectures to enable better prediction performance.
• Score: 4.276883061502341
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While deep learning has revolutionized research and applications in NLP and computer vision, this has not yet been the case for behavioral modeling and behavioral health applications. This is because the domain's datasets are smaller, have heterogeneous datatypes, and typically exhibit a large degree of missingness. Therefore, off-the-shelf deep learning models require significant, often prohibitive, adaptation. Accordingly, many research applications still rely on manually coded features with boosted tree models, sometimes with task-specific features handcrafted by experts. Here, we address these challenges by providing a neural architecture framework for mobile sensing data that can learn generalizable feature representations from time series and demonstrates the feasibility of transfer learning on small data domains through finetuning. This architecture combines benefits from CNN and Trans-former architectures to (1) enable better prediction performance by learning directly from raw minute-level sensor data without the need for handcrafted features by up to 0.33 ROC AUC, and (2) use pretraining to outperform simpler neural models and boosted decision trees with data from as few a dozen participants.

Related papers

• Diffusion-based Neural Network Weights Generation [85.6725307453325]
  We propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• The Trifecta: Three simple techniques for training deeper Forward-Forward networks [0.0]
  We propose a collection of three techniques that synergize exceptionally well and drastically improve the Forward-Forward algorithm on deeper networks. Our experiments demonstrate that our models are on par with similarly structured, backpropagation-based models in both training speed and test accuracy on simple datasets.
  arXiv  Detail & Related papers  (2023-11-29T22:44:32Z)
• Homological Convolutional Neural Networks [4.615338063719135]
  We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations. We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
  arXiv  Detail & Related papers  (2023-08-26T08:48:51Z)
• Efficient Model Adaptation for Continual Learning at the Edge [15.334881190102895]
  Most machine learning (ML) systems assume stationary and matching data distributions during training and deployment. Data distributions often shift over time due to changes in environmental factors, sensor characteristics, and task-of-interest. This paper presents theAdaptor-Reconfigurator (EAR) framework for efficient continual learning under domain shifts.
  arXiv  Detail & Related papers  (2023-08-03T23:55:17Z)
• NCART: Neural Classification and Regression Tree for Tabular Data [0.5439020425819]
  NCART is a modified version of Residual Networks that replaces fully-connected layers with multiple differentiable oblivious decision trees. It maintains its interpretability while benefiting from the end-to-end capabilities of neural networks. The simplicity of the NCART architecture makes it well-suited for datasets of varying sizes.
  arXiv  Detail & Related papers  (2023-07-23T01:27:26Z)
• Neural Attentive Circuits [93.95502541529115]
  We introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) NACs learn the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs achieve an 8x speedup at inference time while losing less than 3% performance.
  arXiv  Detail & Related papers  (2022-10-14T18:00:07Z)
• Transfer Learning with Deep Tabular Models [66.67017691983182]
  We show that upstream data gives tabular neural networks a decisive advantage over GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning. We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
  arXiv  Detail & Related papers  (2022-06-30T14:24:32Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• Inducing Gaussian Process Networks [80.40892394020797]
  We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains. We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-21T05:27:09Z)
• Evaluating deep transfer learning for whole-brain cognitive decoding [11.898286908882561]
  Transfer learning (TL) is well-suited to improve the performance of deep learning (DL) models in datasets with small numbers of samples. Here, we evaluate TL for the application of DL models to the decoding of cognitive states from whole-brain functional Magnetic Resonance Imaging (fMRI) data.
  arXiv  Detail & Related papers  (2021-11-01T15:44:49Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.