Don't stop the training: continuously-updating self-supervised
algorithms best account for auditory responses in the cortex
- URL: http://arxiv.org/abs/2202.07290v1
- Date: Tue, 15 Feb 2022 10:12:56 GMT
- Title: Don't stop the training: continuously-updating self-supervised
algorithms best account for auditory responses in the cortex
- Authors: Pierre Orhan, Yves Boubenec, Jean-R\'emi King
- Abstract summary: We analyze the brain responses of two ferret auditory cortices recorded with functional UltraSound imaging (fUS)
We compare these brain responses to the activations of Wav2vec 2.0, a self-supervised neural network pretrained with 960,h of speech.
- Score: 1.7725414095035827
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Over the last decade, numerous studies have shown that deep neural networks
exhibit sensory representations similar to those of the mammalian brain, in
that their activations linearly map onto cortical responses to the same sensory
inputs. However, it remains unknown whether these artificial networks also
learn like the brain. To address this issue, we analyze the brain responses of
two ferret auditory cortices recorded with functional UltraSound imaging (fUS),
while the animals were presented with 320 10\,s sounds. We compare these brain
responses to the activations of Wav2vec 2.0, a self-supervised neural network
pretrained with 960\,h of speech, and input with the same 320 sounds.
Critically, we evaluate Wav2vec 2.0 under two distinct modes: (i) "Pretrained",
where the same model is used for all sounds, and (ii) "Continuous Update",
where the weights of the pretrained model are modified with back-propagation
after every sound, presented in the same order as the ferrets. Our results show
that the Continuous-Update mode leads Wav2Vec 2.0 to generate activations that
are more similar to the brain than a Pretrained Wav2Vec 2.0 or than other
control models using different training modes. These results suggest that the
trial-by-trial modifications of self-supervised algorithms induced by
back-propagation aligns with the corresponding fluctuations of cortical
responses to sounds. Our finding thus provides empirical evidence of a common
learning mechanism between self-supervised models and the mammalian cortex
during sound processing.
Related papers
- Teaching Wav2Vec2 the Language of the Brain [13.094509587996082]
We show that patterns learned by Wav2Vec2 are transferable to brain data.
We execute full fine-tuning with pre-trained weights for Wav2Vec2, training ''from scratch'' without pre-trained weights as well as freezing a pre-trained Wav2Vec2 and training only the BFE each for 45 different BFE architectures.
arXiv Detail & Related papers (2025-01-16T10:37:07Z) - Utilizing Machine Learning and 3D Neuroimaging to Predict Hearing Loss: A Comparative Analysis of Dimensionality Reduction and Regression Techniques [0.0]
We have explored machine learning approaches for predicting hearing loss thresholds on the brain's gray matter 3D images.
In the first phase, we used a 3D CNN model to reduce high-dimensional input into latent space.
In the second phase, we utilized this model to reduce input into rich features.
arXiv Detail & Related papers (2024-04-30T18:39:41Z) - An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits [22.558134249701794]
We propose a novel cortico-thalamo-cortical neural network (CTCNet) for audio-visual speech separation (AVSS)
CTCNet learns hierarchical auditory and visual representations in a bottom-up manner in separate auditory and visualworks.
Experiments on three speech separation benchmark datasets show that CTCNet remarkably outperforms existing AVSS methods with considerably fewer parameters.
arXiv Detail & Related papers (2022-12-21T03:28:30Z) - Jointly Learning Visual and Auditory Speech Representations from Raw
Data [108.68531445641769]
RAVEn is a self-supervised multi-modal approach to jointly learn visual and auditory speech representations.
Our design is asymmetric w.r.t. driven by the inherent differences between video and audio.
RAVEn surpasses all self-supervised methods on visual speech recognition.
arXiv Detail & Related papers (2022-12-12T21:04:06Z) - Neural Language Models are not Born Equal to Fit Brain Data, but
Training Helps [75.84770193489639]
We examine the impact of test loss, training corpus and model architecture on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook.
We find that untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words.
We suggest good practices for future studies aiming at explaining the human language system using neural language models.
arXiv Detail & Related papers (2022-07-07T15:37:17Z) - Toward a realistic model of speech processing in the brain with
self-supervised learning [67.7130239674153]
Self-supervised algorithms trained on the raw waveform constitute a promising candidate.
We show that Wav2Vec 2.0 learns brain-like representations with as little as 600 hours of unlabelled speech.
arXiv Detail & Related papers (2022-06-03T17:01:46Z) - Self-supervised models of audio effectively explain human cortical
responses to speech [71.57870452667369]
We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system.
We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
arXiv Detail & Related papers (2022-05-27T22:04:02Z) - Learning Personal Representations from fMRIby Predicting Neurofeedback
Performance [52.77024349608834]
We present a deep neural network method for learning a personal representation for individuals performing a self neuromodulation task, guided by functional MRI (fMRI)
The representation is learned by a self-supervised recurrent neural network, that predicts the Amygdala activity in the next fMRI frame given recent fMRI frames and is conditioned on the learned individual representation.
arXiv Detail & Related papers (2021-12-06T10:16:54Z) - Artificial Neural Variability for Deep Learning: On Overfitting, Noise
Memorization, and Catastrophic Forgetting [135.0863818867184]
artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks.
ANV plays as an implicit regularizer of the mutual information between the training data and the learned model.
It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
arXiv Detail & Related papers (2020-11-12T06:06:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.