Range, not Independence, Drives Modularity in Biologically Inspired Representations

• URL: http://arxiv.org/abs/2410.06232v4
• Date: Fri, 11 Apr 2025 14:14:17 GMT
• Title: Range, not Independence, Drives Modularity in Biologically Inspired Representations
• Authors: Will Dorrell, Kyle Hsu, Luke Hollingsworth, Jin Hwa Lee, Jiajun Wu, Chelsea Finn, Peter E Latham, Tim EJ Behrens, James CR Whittington,
• Abstract summary: We develop a theory of when biologically inspired networks modularise their representation of source variables (sources)<n>We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal linear autoencoder modularise.<n>Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work.
• Score: 52.48094670415497
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired networks -- those that are nonnegative and energy efficient -- modularise their representation of source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather we show that sources modularise if their support is ``sufficiently spread''. From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data, showing that range independence can be used to understand the mixing or modularising of spatial and reward information in entorhinal recordings in seemingly conflicting experiments. Further, we use these results to suggest alternate origins of mixed-selectivity, beyond the predominant theory of flexible nonlinear classification. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.

Related papers

• Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors [5.967290675400836]
  We propose a probabilistic model that predicts the joint distribution of the neuronal responses from video stimuli and stimulus-independent latent factors. We find that it outperforms video-only models in terms of log-likelihood and achieves improvements in likelihood and correlation when conditioned on responses from other neurons.
  arXiv  Detail & Related papers  (2024-10-21T16:01:39Z)
• A generative framework to bridge data-driven models and scientific theories in language neuroscience [84.76462599023802]
  We present generative explanation-mediated validation, a framework for generating concise explanations of language selectivity in the brain. We show that explanatory accuracy is closely related to the predictive power and stability of the underlying statistical models.
  arXiv  Detail & Related papers  (2024-10-01T15:57:48Z)
• Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data [3.46029409929709]
  State-of-the-art systems neuroscience experiments yield large-scale multimodal data, and these data sets require new tools for analysis. Inspired by the success of large pretrained models in vision and language domains, we reframe the analysis of large-scale, cellular-resolution neuronal spiking data into an autoregressive generation problem. We first trained Neuroformer on simulated datasets, and found that it both accurately predicted intrinsically simulated neuronal circuit activity, and also inferred the underlying neural circuit connectivity, including direction.
  arXiv  Detail & Related papers  (2023-10-31T20:17:32Z)
• Modular Boundaries in Recurrent Neural Networks [39.626497874552555]
  We use a community detection method from network science known as modularity to partition neurons into distinct modules. These partitions allow us to ask the following question: do these modular boundaries matter to the system?
  arXiv  Detail & Related papers  (2023-10-31T16:37:01Z)
• Inferring Inference [7.11780383076327]
  We develop a framework for inferring canonical distributed computations from large-scale neural activity patterns. We simulate recordings for a model brain that implicitly implements an approximate inference algorithm on a probabilistic graphical model. Overall, this framework provides a new tool for discovering interpretable structure in neural recordings.
  arXiv  Detail & Related papers  (2023-10-04T22:12:11Z)
• Emergent Modularity in Pre-trained Transformers [127.08792763817496]
  We consider two main characteristics of modularity: functional specialization of neurons and function-based neuron grouping. We study how modularity emerges during pre-training, and find that the modular structure is stabilized at the early stage. It suggests that Transformers first construct the modular structure and then learn fine-grained neuron functions.
  arXiv  Detail & Related papers  (2023-05-28T11:02:32Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• EINNs: Epidemiologically-Informed Neural Networks [75.34199997857341]
  We introduce a new class of physics-informed neural networks-EINN-crafted for epidemic forecasting. We investigate how to leverage both the theoretical flexibility provided by mechanistic models as well as the data-driven expressability afforded by AI models.
  arXiv  Detail & Related papers  (2022-02-21T18:59:03Z)
• Modeling Implicit Bias with Fuzzy Cognitive Maps [0.0]
  This paper presents a Fuzzy Cognitive Map model to quantify implicit bias in structured datasets. We introduce a new reasoning mechanism equipped with a normalization-like transfer function that prevents neurons from saturating.
  arXiv  Detail & Related papers  (2021-12-23T17:04:12Z)
• Logical Information Cells I [10.411800812671952]
  In this study we explore the spontaneous apparition of visible intelligible reasoning in simple artificial networks. We start with the reproduction of a DNN model of natural neurons in monkeys. We then study a bit more complex tasks, a priori involving predicate logic.
  arXiv  Detail & Related papers  (2021-08-10T15:31:26Z)
• The Causal Neural Connection: Expressiveness, Learnability, and Inference [125.57815987218756]
  An object called structural causal model (SCM) represents a collection of mechanisms and sources of random variation of the system under investigation. In this paper, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020) still holds for neural models. We introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences.
  arXiv  Detail & Related papers  (2021-07-02T01:55:18Z)
• The Neural Coding Framework for Learning Generative Models [91.0357317238509]
  We propose a novel neural generative model inspired by the theory of predictive processing in the brain. In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
  arXiv  Detail & Related papers  (2020-12-07T01:20:38Z)
• TaBooN -- Boolean Network Synthesis Based on Tabu Search [0.0]
  Omics-technologies revolutionized the investigation of biology by producing molecular data in multiple dimensions and scale. Biological network is composed of nodes referring to the components such as genes or proteins, and the edges/arcs formalizing interactions between them.
  arXiv  Detail & Related papers  (2020-09-08T08:56:14Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.