Information theoretic study of the neural geometry induced by category learning

• URL: http://arxiv.org/abs/2311.15682v1
• Date: Mon, 27 Nov 2023 10:16:22 GMT
• Title: Information theoretic study of the neural geometry induced by category learning
• Authors: Laurent Bonnasse-Gahot and Jean-Pierre Nadal
• Abstract summary: We take an information theoretic approach to assess the efficiency of the representations induced by category learning. One main consequence is that category learning induces an expansion of neural space near decision boundaries.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Categorization is an important topic both for biological and artificial neural networks. Here, we take an information theoretic approach to assess the efficiency of the representations induced by category learning. We show that one can decompose the relevant Bayesian cost into two components, one for the coding part and one for the decoding part. Minimizing the coding cost implies maximizing the mutual information between the set of categories and the neural activities. We analytically show that this mutual information can be written as the sum of two terms that can be interpreted as (i) finding an appropriate representation space, and, (ii) building a representation with the appropriate metrics, based on the neural Fisher information on this space. One main consequence is that category learning induces an expansion of neural space near decision boundaries. Finally, we provide numerical illustrations that show how Fisher information of the coding neural population aligns with the boundaries between categories.

Related papers

• SimNP: Learning Self-Similarity Priors Between Neural Points [52.4201466988562]
  SimNP is a method to learn category-level self-similarities. We show that SimNP is able to outperform previous methods in reconstructing symmetric unseen object regions.
  arXiv  Detail & Related papers  (2023-09-07T16:02:40Z)
• Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognition [71.35205097460124]
  How humans understand and recognize the actions of others is a complex neuroscientific problem. LA-GCN proposes a graph convolution network using large-scale language models (LLM) knowledge assistance.
  arXiv  Detail & Related papers  (2023-05-21T08:29:16Z)
• Graph Neural Operators for Classification of Spatial Transcriptomics Data [1.408706290287121]
  We propose a study incorporating various graph neural network approaches to validate the efficacy of applying neural operators towards prediction of brain regions in mouse brain tissue samples. We were able to achieve an F1 score of nearly 72% for the graph neural operator approach which outperformed all baseline and other graph network approaches.
  arXiv  Detail & Related papers  (2023-02-01T18:32:06Z)
• Higher-order mutual information reveals synergistic sub-networks for multi-neuron importance [0.483420384410068]
  Previous work primarily attributed importance to individual neurons. In this work, we study which groups of neurons contain synergistic or redundant information. Results suggest our method can be used for pruning and unsupervised representation learning.
  arXiv  Detail & Related papers  (2022-11-01T12:21:15Z)
• Invariant Representations with Stochastically Quantized Neural Networks [5.7923858184309385]
  We propose a methodology for direct computation of the mutual information between a neural layer and a sensitive attribute. We show that this method compares favorably with the state of the art in fair representation learning.
  arXiv  Detail & Related papers  (2022-08-04T13:36:06Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• The Causal Neural Connection: Expressiveness, Learnability, and Inference [125.57815987218756]
  An object called structural causal model (SCM) represents a collection of mechanisms and sources of random variation of the system under investigation. In this paper, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020) still holds for neural models. We introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences.
  arXiv  Detail & Related papers  (2021-07-02T01:55:18Z)
• Generalized Organ Segmentation by Imitating One-shot Reasoning using Anatomical Correlation [55.1248480381153]
  We propose OrganNet which learns a generalized organ concept from a set of annotated organ classes and then transfer this concept to unseen classes. We show that OrganNet can effectively resist the wide variations in organ morphology and produce state-of-the-art results in one-shot segmentation task.
  arXiv  Detail & Related papers  (2021-03-30T13:41:12Z)
• Interpretable Neural Networks based classifiers for categorical inputs [0.0]
  We introduce a simple way to interpret the output function of a neural network classifier that take as input categorical variables. We show that in these cases each layer of the network, and the logits layer in particular, can be expanded as a sum of terms that account for the contribution to the classification of each input pattern. The analysis of the contributions of each pattern, after an appropriate gauge transformation, is presented in two cases where the effectiveness of the method can be appreciated.
  arXiv  Detail & Related papers  (2021-02-05T14:38:50Z)
• Categorical Perception: A Groundwork for Deep Learning [0.0]
  We study categorical effects in artificial neural networks. We show on both shallow and deep neural networks that category learning automatically induces categorical perception. An important outcome of our analysis is to provide a coherent and unifying view of the efficacy of the dropout regularization technique.
  arXiv  Detail & Related papers  (2020-12-10T09:41:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.