Sleep-Like Unsupervised Replay Improves Performance when Data are Limited or Unbalanced

• URL: http://arxiv.org/abs/2402.10956v1
• Date: Mon, 12 Feb 2024 18:25:41 GMT
• Title: Sleep-Like Unsupervised Replay Improves Performance when Data are Limited or Unbalanced
• Abstract summary: Sleep was implemented as an unsupervised phase with local Hebbian type learning rules. We found a significant boost in accuracy after the sleep phase for models trained with limited data in the range of 0.5-10% of total MNIST or Fashion MNIST datasets. This study sheds light on a potential synaptic weight dynamics strategy employed by the brain during sleep to enhance memory performance when training data are limited or imbalanced.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The performance of artificial neural networks (ANNs) degrades when training data are limited or imbalanced. In contrast, the human brain can learn quickly from just a few examples. Here, we investigated the role of sleep in improving the performance of ANNs trained with limited data on the MNIST and Fashion MNIST datasets. Sleep was implemented as an unsupervised phase with local Hebbian type learning rules. We found a significant boost in accuracy after the sleep phase for models trained with limited data in the range of 0.5-10% of total MNIST or Fashion MNIST datasets. When more than 10% of the total data was used, sleep alone had a slight negative impact on performance, but this was remedied by fine-tuning on the original data. This study sheds light on a potential synaptic weight dynamics strategy employed by the brain during sleep to enhance memory performance when training data are limited or imbalanced.

Related papers

• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations [46.963298896594814]
  Training Data Attribution seeks to trace a model's predictions back to its training data.<n>Most approaches approximate this effect in the parameter space using gradients.<n>We model the functional effect of training data in the activation space.
  arXiv  Detail & Related papers  (2026-06-03T17:59:36Z)
• Forget-MI: Machine Unlearning for Forgetting Multimodal Information in Healthcare Settings [5.200386658850142]
  Forget-MI is a novel machine unlearning method for multimodal medical data.<n>We evaluate our results using performance on the forget dataset, performance on the test dataset, and Membership Inference Attack (MIA)<n>Our approach reduces MIA by 0.202 and decreases AUC and F1 scores on the forget set by 0.221 and 0.305, respectively.
  arXiv  Detail & Related papers  (2025-06-29T08:53:23Z)
• PSDNorm: Test-Time Temporal Normalization for Deep Learning in Sleep Staging [63.05435596565677]
  We propose PSDNorm that leverages Monge mapping and temporal context to normalize feature maps in deep learning models for signals.<n> PSDNorm achieves state-of-the-art performance on unseen left-out datasets while being 4-times more data-efficient than BatchNorm.
  arXiv  Detail & Related papers  (2025-03-06T16:20:25Z)
• Unsupervised Replay Strategies for Continual Learning with Limited Data [1.0687104237121408]
  We show that introducing a sleep phase significantly enhanced accuracy in models trained with limited data. When a few tasks were trained sequentially, sleep replay rescued previously learned information that had been catastrophically forgetting following new task training. This study highlights the multifaceted role of sleep replay in augmenting learning efficiency and facilitating continual learning in ANNs.
  arXiv  Detail & Related papers  (2024-10-21T16:21:09Z)
• Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment [126.34547428473968]
  Large language models (LLMs) are still struggling in aligning with human preference in complex tasks and scenarios. We propose a low-redundant alignment method named textbfALLO, focusing on optimizing the most related neurons with the most useful supervised signals. Experimental results on 10 datasets have shown the effectiveness of ALLO.
  arXiv  Detail & Related papers  (2024-06-18T13:34:40Z)
• Label-Efficient Sleep Staging Using Transformers Pre-trained with Position Prediction [2.591936982899312]
  We propose an architecture that seamlessly couples the feature and temporal encoding and a suitable pretraining scheme that pretrains the entire model. On a sample sleep staging dataset, we find that the proposed scheme offers performance gains that do not saturate with amount of labeled training data.
  arXiv  Detail & Related papers  (2024-03-29T23:22:30Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Convolutional Monge Mapping Normalization for learning on sleep data [63.22081662149488]
  We propose a new method called Convolutional Monge Mapping Normalization (CMMN) CMMN consists in filtering the signals in order to adapt their power spectrum density (PSD) to a Wasserstein barycenter estimated on training data. Numerical experiments on sleep EEG data show that CMMN leads to significant and consistent performance gains independent from the neural network architecture.
  arXiv  Detail & Related papers  (2023-05-30T08:24:01Z)
• Self-Supervised Mental Disorder Classifiers via Time Reversal [0.0]
  We demonstrate that a model trained on the time direction of functional neuro-imaging data could help in any downstream task. We train a Deep Neural Network on Independent components derived from fMRI data using the Independent component analysis (ICA) technique. We show that learning time direction helps a model learn some causal relation in fMRI data that helps in faster convergence.
  arXiv  Detail & Related papers  (2022-11-29T17:24:43Z)
• Continual learning benefits from multiple sleep mechanisms: NREM, REM, and Synaptic Downscaling [51.316408685035526]
  Learning new tasks and skills in succession without losing prior learning is a computational challenge for both artificial and biological neural networks. Here, we investigate how modeling three distinct components of mammalian sleep together affects continual learning in artificial neural networks.
  arXiv  Detail & Related papers  (2022-09-09T13:45:27Z)
• Neurochaos Feature Transformation and Classification for Imbalanced Learning [0.0]
  Learning from limited and imbalanced data is a challenging problem in the Artificial Intelligence community. Inspired by the chaotic neuronal firing in the human brain, a novel learning algorithm namely Neurochaos Learning (NL) was recently proposed. We propose a unique combination of neurochaos based feature transformation and extraction with traditional ML algorithms.
  arXiv  Detail & Related papers  (2022-04-20T16:11:45Z)
• Variational voxelwise rs-fMRI representation learning: Evaluation of sex, age, and neuropsychiatric signatures [0.0]
  We propose to apply non-linear representation learning to voxelwise rs-fMRI data. Learning the non-linear representations is done using a variational autoencoder (VAE) VAE is trained on voxelwise rs-fMRI data and performs non-linear dimensionality reduction that retains meaningful information.
  arXiv  Detail & Related papers  (2021-08-29T05:27:32Z)
• Deep Time Delay Neural Network for Speech Enhancement with Full Data Learning [60.20150317299749]
  This paper proposes a deep time delay neural network (TDNN) for speech enhancement with full data learning. To make full use of the training data, we propose a full data learning method for speech enhancement.
  arXiv  Detail & Related papers  (2020-11-11T06:32:37Z)
• Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation [86.40610684026262]
  In this work, we explore to identify the inactive training examples which contribute less to the model performance. We introduce data rejuvenation to improve the training of NMT models on large-scale datasets by exploiting inactive examples. Experimental results on WMT14 English-German and English-French datasets show that the proposed data rejuvenation consistently and significantly improves performance for several strong NMT models.
  arXiv  Detail & Related papers  (2020-10-06T08:57:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.