Neural Encoding and Decoding at Scale

• URL: http://arxiv.org/abs/2504.08201v3
• Date: Sun, 20 Apr 2025 20:44:18 GMT
• Title: Neural Encoding and Decoding at Scale
• Authors: Yizi Zhang, Yanchen Wang, Mehdi Azabou, Alexandre Andre, Zixuan Wang, Hanrui Lyu, The International Brain Laboratory, Eva Dyer, Liam Paninski, Cole Hurwitz,
• Abstract summary: We introduce a multimodal, multi-task model that enables simultaneous Neural and Decoding at Scale (NEDS)<n>Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking.<n>NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals.
• Score: 42.33285735011587
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has demonstrated that large-scale, multi-animal models are powerful tools for characterizing the relationship between neural activity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or predicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behavior. To bridge this gap, we introduce a multimodal, multi-task model that enables simultaneous Neural Encoding and Decoding at Scale (NEDS). Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 animals performing the same visual decision-making task. In comparison to other large-scale models, we demonstrate that NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surprisingly, NEDS's learned embeddings exhibit emergent properties: even without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model of the brain that enables seamless translation between neural activity and behavior.

Related papers

• QuantFormer: Learning to Quantize for Neural Activity Forecasting in Mouse Visual Cortex [26.499583552980248]
  QuantFormer is a transformer-based model specifically designed for forecasting neural activity from two-photon calcium imaging data.<n> QuantFormer sets a new benchmark in forecasting mouse visual cortex activity.<n>It demonstrates robust performance and generalization across various stimuli and individuals.
  arXiv  Detail & Related papers  (2024-12-10T07:44:35Z)
• Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models [2.600709013150986]
  Understanding the neural basis of behavior is a fundamental goal in neuroscience. Our approach, named BeNeDiff'', first identifies a fine-grained and disentangled neural subspace. It then employs state-of-the-art generative diffusion models to synthesize behavior videos that interpret the neural dynamics of each latent factor.
  arXiv  Detail & Related papers  (2024-10-12T18:28:56Z)
• Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution [10.49121904052395]
  We build towards a first foundation model for neural spiking data that can solve a diverse set of tasks across multiple brain areas. Prediction tasks include single-neuron and region-level activity prediction, forward prediction, and behavior decoding.
  arXiv  Detail & Related papers  (2024-07-19T21:05:28Z)
• A Unified, Scalable Framework for Neural Population Decoding [12.052847252465826]
  We introduce a training framework and architecture designed to model the population dynamics of neural activity. We construct a large-scale multi-session model trained on large datasets from seven nonhuman primates.
  arXiv  Detail & Related papers  (2023-10-24T17:58:26Z)
• Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps [75.84770193489639]
  We examine the impact of test loss, training corpus and model architecture on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words. We suggest good practices for future studies aiming at explaining the human language system using neural language models.
  arXiv  Detail & Related papers  (2022-07-07T15:37:17Z)
• Overcoming the Domain Gap in Neural Action Representations [60.47807856873544]
  3D pose data can now be reliably extracted from multi-view video sequences without manual intervention. We propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions.
  arXiv  Detail & Related papers  (2021-12-02T12:45:46Z)
• Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations [60.47807856873544]
  A fundamental goal in neuroscience is to understand the relationship between neural activity and behavior. We generated a new multimodal dataset consisting of the spontaneous behaviors generated by fruit flies. This dataset and our new set of augmentations promise to accelerate the application of self-supervised learning methods in neuroscience.
  arXiv  Detail & Related papers  (2021-11-29T15:27:51Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.