Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces

• URL: http://arxiv.org/abs/2505.07831v1
• Date: Wed, 30 Apr 2025 12:33:28 GMT
• Title: Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces
• Authors: Michael Pichat, William Pogrund, Paloma Pichat, Judicael Poumay, Armanouche Gasparian, Samuel Demarchi, Martin Corbet, Alois Georgeon, Michael Veillet-Guillem,
• Abstract summary: We geometrically define a neuron in layer n as a categorical vector space with a non-orthogonal basis, composed of categorical sub-dimensions extracted from preceding neurons in layer n-1.<n>This categorical vector space is structured by the activation space of each neuron and enables, via an intra-neuronal attention process, the identification and utilization of a critical categorical zone for the efficiency of the language model.
• Score: 0.11608974088441382
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The polysemantic nature of synthetic neurons in artificial intelligence language models is currently understood as the result of a necessary superposition of distributed features within the latent space. We propose an alternative approach, geometrically defining a neuron in layer n as a categorical vector space with a non-orthogonal basis, composed of categorical sub-dimensions extracted from preceding neurons in layer n-1. This categorical vector space is structured by the activation space of each neuron and enables, via an intra-neuronal attention process, the identification and utilization of a critical categorical zone for the efficiency of the language model - more homogeneous and located at the intersection of these different categorical sub-dimensions.

Related papers

• Intra-neuronal attention within language models Relationships between activation and semantics [0.0]
  This study investigates the ability of perceptron-type neurons in language models to perform intra-neuronal attention.<n>The objective of this work is to determine to what extent formal neurons can establish a homomorphic relationship between activation-based and categorical segmentations.
  arXiv  Detail & Related papers  (2025-03-17T09:47:11Z)
• Neuron Platonic Intrinsic Representation From Dynamics Using Contrastive Learning [4.538151592901714]
  We propose the NeurPIR (Neuron Platonic Intrinsic Representation) framework.<n>It uses contrastive learning, with segments from the same neuron as positive pairs and those from different neurons as negative pairs.<n>We tested our method on Izhikevich model-simulated neuronal population dynamics data.
  arXiv  Detail & Related papers  (2025-02-06T02:22:23Z)
• The Process of Categorical Clipping at the Core of the Genesis of Concepts in Synthetic Neural Cognition [0.0]
  This article investigates, within the field of neuropsychology of artificial intelligence, the process of categorical segmentation performed by language models.<n>This process involves, across different neural layers, the creation of new functional categorical dimensions to analyze the input textual data and perform the required tasks.<n>We explore several cognitive characteristics of this synthetic clipping in an exploratory manner.
  arXiv  Detail & Related papers  (2025-01-21T11:32:39Z)
• Neuropsychology and Explainability of AI: A Distributional Approach to the Relationship Between Activation Similarity of Neural Categories in Synthetic Cognition [0.11235145048383502]
  We propose an approach to explainability of artificial neural networks that involves using concepts from human cognitive tokens. We show that the categorical segment created by a neuron is actually the result of a superposition of categorical sub-dimensions within its input vector space.
  arXiv  Detail & Related papers  (2024-10-23T05:27:09Z)
• Multilayer Multiset Neuronal Networks -- MMNNs [55.2480439325792]
  The present work describes multilayer multiset neuronal networks incorporating two or more layers of coincidence similarity neurons. The work also explores the utilization of counter-prototype points, which are assigned to the image regions to be avoided.
  arXiv  Detail & Related papers  (2023-08-28T12:55:13Z)
• Self-Supervised Graph Representation Learning for Neuronal Morphologies [75.38832711445421]
  We present GraphDINO, a data-driven approach to learn low-dimensional representations of 3D neuronal morphologies from unlabeled datasets. We show, in two different species and across multiple brain areas, that this method yields morphological cell type clusterings on par with manual feature-based classification by experts. Our method could potentially enable data-driven discovery of novel morphological features and cell types in large-scale datasets.
  arXiv  Detail & Related papers  (2021-12-23T12:17:47Z)
• Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders [72.15087604017441]
  Category-selectivity describes the observation that certain spatially localized areas of the cerebral cortex tend to respond robustly and selectively to stimuli from specific limited categories. We leverage the newly introduced Topographic Variational Autoencoder to model of the emergence of such localized category-selectivity in an unsupervised manner. We show preliminary results suggesting that our model yields a nested spatial hierarchy of increasingly abstract categories, analogous to observations from the human ventral temporal cortex.
  arXiv  Detail & Related papers  (2021-10-25T11:37:41Z)
• Complexity-based speciation and genotype representation for neuroevolution [81.21462458089142]
  This paper introduces a speciation principle for neuroevolution where evolving networks are grouped into species based on the number of hidden neurons. The proposed speciation principle is employed in several techniques designed to promote and preserve diversity within species and in the ecosystem as a whole.
  arXiv  Detail & Related papers  (2020-10-11T06:26:56Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Hyperbolic Neural Networks++ [66.16106727715061]
  We generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincar'e ball model. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.
  arXiv  Detail & Related papers  (2020-06-15T08:23:20Z)
• Emergence of Separable Manifolds in Deep Language Representations [26.002842878797765]
  Deep neural networks (DNNs) have shown much empirical success in solving perceptual tasks across various cognitive modalities. Recent studies report considerable similarities between representations extracted from task-optimized DNNs and neural populations in the brain. DNNs have subsequently become a popular model class to infer computational principles underlying complex cognitive functions.
  arXiv  Detail & Related papers  (2020-06-01T17:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.