Disentanglement with Biological Constraints: A Theory of Functional Cell Types

• URL: http://arxiv.org/abs/2210.01768v2
• Date: Fri, 31 Mar 2023 18:41:15 GMT
• Title: Disentanglement with Biological Constraints: A Theory of Functional Cell Types
• Authors: James C.R. Whittington, Will Dorrell, Surya Ganguli, Timothy E.J. Behrens
• Abstract summary: This work provides a mathematical understanding of why single neurons in the brain often represent single human-interpretable factors. It also steps towards an understanding task structure shapes the structure of brain representation.
• Score: 20.929056085868613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neurons in the brain are often finely tuned for specific task variables. Moreover, such disentangled representations are highly sought after in machine learning. Here we mathematically prove that simple biological constraints on neurons, namely nonnegativity and energy efficiency in both activity and weights, promote such sought after disentangled representations by enforcing neurons to become selective for single factors of task variation. We demonstrate these constraints lead to disentanglement in a variety of tasks and architectures, including variational autoencoders. We also use this theory to explain why the brain partitions its cells into distinct cell types such as grid and object-vector cells, and also explain when the brain instead entangles representations in response to entangled task factors. Overall, this work provides a mathematical understanding of why single neurons in the brain often represent single human-interpretable factors, and steps towards an understanding task structure shapes the structure of brain representation.

Related papers

• How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neurons [0.0]
  How do the synthetic neurons in language models create "thought" categories to segment and analyze their informational environment? This study explores these concepts through the notions of priming, attention, and categorical phasing.
  arXiv  Detail & Related papers  (2024-12-26T16:26:00Z)
• Brain-like Functional Organization within Large Language Models [58.93629121400745]
  The human brain has long inspired the pursuit of artificial intelligence (AI) Recent neuroimaging studies provide compelling evidence of alignment between the computational representation of artificial neural networks (ANNs) and the neural responses of the human brain to stimuli. In this study, we bridge this gap by directly coupling sub-groups of artificial neurons with functional brain networks (FBNs) This framework links the AN sub-groups to FBNs, enabling the delineation of brain-like functional organization within large language models (LLMs)
  arXiv  Detail & Related papers  (2024-10-25T13:15:17Z)
• Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment [65.268245109828]
  We examine algorithms for conducting credit assignment in artificial neural networks that are inspired or motivated by neurobiology. We organize the ever-growing set of brain-inspired learning schemes into six general families and consider these in the context of backpropagation of errors. The results of this review are meant to encourage future developments in neuro-mimetic systems and their constituent learning processes.
  arXiv  Detail & Related papers  (2023-12-01T05:20:57Z)
• A Goal-Driven Approach to Systems Neuroscience [2.6451153531057985]
  Humans and animals exhibit a range of interesting behaviors in dynamic environments. It is unclear how our brains actively reformat this dense sensory information to enable these behaviors. We offer a new definition of interpretability that we show has promise in yielding unified structural and functional models of neural circuits.
  arXiv  Detail & Related papers  (2023-11-05T16:37:53Z)
• Single Biological Neurons as Temporally Precise Spatio-Temporal Pattern Recognizers [0.0]
  thesis is focused on the central idea that single neurons in the brain should be regarded as temporally highly complex-temporal pattern recognizers. In chapter 2 we demonstrate that single neurons can generate temporally precise output patterns in response to specific-temporal input patterns. In chapter 3, we use the differentiable deep network of a realistic cortical neuron as a tool to approximate the implications of the output of the neuron.
  arXiv  Detail & Related papers  (2023-09-26T17:32:08Z)
• Neuronal Cell Type Classification using Deep Learning [3.3517146652431378]
  Recent developments in machine learning have provided advanced abilities for classifying neurons. This paper aims to provide a robust and explainable deep-learning framework to classify neurons based on their electrophysiological activity.
  arXiv  Detail & Related papers  (2023-06-01T10:28:49Z)
• Emergent Modularity in Pre-trained Transformers [127.08792763817496]
  We consider two main characteristics of modularity: functional specialization of neurons and function-based neuron grouping. We study how modularity emerges during pre-training, and find that the modular structure is stabilized at the early stage. It suggests that Transformers first construct the modular structure and then learn fine-grained neuron functions.
  arXiv  Detail & Related papers  (2023-05-28T11:02:32Z)
• Constraints on the design of neuromorphic circuits set by the properties of neural population codes [61.15277741147157]
  In the brain, information is encoded, transmitted and used to inform behaviour. Neuromorphic circuits need to encode information in a way compatible to that used by populations of neuron in the brain.
  arXiv  Detail & Related papers  (2022-12-08T15:16:04Z)
• Explanatory models in neuroscience: Part 2 -- constraint-based intelligibility [8.477619837043214]
  Computational modeling plays an increasingly important role in neuroscience, highlighting the philosophical question of how models explain. In biological systems, many of these dependencies are naturally "top-down" We show how the optimization techniques used to construct NN models capture some key aspects of these dependencies.
  arXiv  Detail & Related papers  (2021-04-03T22:14:01Z)
• The distribution of inhibitory neurons in the C. elegans connectome facilitates self-optimization of coordinated neural activity [78.15296214629433]
  The nervous system of the nematode Caenorhabditis elegans exhibits remarkable complexity despite the worm's small size. A general challenge is to better understand the relationship between neural organization and neural activity at the system level. We implemented an abstract simulation model of the C. elegans connectome that approximates the neurotransmitter identity of each neuron.
  arXiv  Detail & Related papers  (2020-10-28T23:11:37Z)
• Compositional Explanations of Neurons [52.71742655312625]
  We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts. We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
  arXiv  Detail & Related papers  (2020-06-24T20:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.