Categorical Perception: A Groundwork for Deep Learning

• URL: http://arxiv.org/abs/2012.05549v1
• Date: Thu, 10 Dec 2020 09:41:38 GMT
• Title: Categorical Perception: A Groundwork for Deep Learning
• Authors: Laurent Bonnasse-Gahot and Jean-Pierre Nadal
• Abstract summary: We study categorical effects in artificial neural networks. We show on both shallow and deep neural networks that category learning automatically induces categorical perception. An important outcome of our analysis is to provide a coherent and unifying view of the efficacy of the dropout regularization technique.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Classification is one of the major tasks that deep learning is successfully tackling. Categorization is also a fundamental cognitive ability. A well-known perceptual consequence of categorization in humans and other animals, called categorical perception, is characterized by a within-category compression and a between-category separation: two items, close in input space, are perceived closer if they belong to the same category than if they belong to different categories. Elaborating on experimental and theoretical results in cognitive science, here we study categorical effects in artificial neural networks. Our formal and numerical analysis provides insights into the geometry of the neural representation in deep layers, with expansion of space near category boundaries and contraction far from category boundaries. We investigate categorical representation by using two complementary approaches: one mimics experiments in psychophysics and cognitive neuroscience by means of morphed continua between stimuli of different categories, while the other introduces a categoricality index that quantifies the separability of the classes at the population level (a given layer in the neural network). We show on both shallow and deep neural networks that category learning automatically induces categorical perception. We further show that the deeper a layer, the stronger the categorical effects. An important outcome of our analysis is to provide a coherent and unifying view of the efficacy of different heuristic practices of the dropout regularization technique. Our views, which find echoes in the neuroscience literature, insist on the differential role of noise as a function of the level of representation and in the course of learning: noise injected in the hidden layers gets structured according to the organization of the categories, more variability being allowed within a category than across classes.

Related papers

• Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy [44.99833362998488]
  We argue that the learning process in classification problems can be understood through the lens of label clustering. Specifically, we observe that networks tend to distinguish higher-level (hypernym) categories in the early stages of training. We introduce a novel framework to track the evolution of the feature manifold during training, revealing how the hierarchy of class relations emerges.
  arXiv  Detail & Related papers  (2025-02-17T18:47:01Z)
• The Process of Categorical Clipping at the Core of the Genesis of Concepts in Synthetic Neural Cognition [0.0]
  This article investigates, within the field of neuropsychology of artificial intelligence, the process of categorical segmentation performed by language models. This process involves, across different neural layers, the creation of new functional categorical dimensions to analyze the input textual data and perform the required tasks. We explore several cognitive characteristics of this synthetic clipping in an exploratory manner.
  arXiv  Detail & Related papers  (2025-01-21T11:32:39Z)
• Neuropsychology and Explainability of AI: A Distributional Approach to the Relationship Between Activation Similarity of Neural Categories in Synthetic Cognition [0.11235145048383502]
  We propose an approach to explainability of artificial neural networks that involves using concepts from human cognitive tokens. We show that the categorical segment created by a neuron is actually the result of a superposition of categorical sub-dimensions within its input vector space.
  arXiv  Detail & Related papers  (2024-10-23T05:27:09Z)
• Understanding the Role of Pathways in a Deep Neural Network [4.456675543894722]
  We analyze a convolutional neural network (CNN) trained in the classification task and present an algorithm to extract the diffusion pathways of individual pixels. We find that the few largest pathways of an individual pixel from an image tend to cross the feature maps in each layer that is important for classification.
  arXiv  Detail & Related papers  (2024-02-28T07:53:19Z)
• Finding Interpretable Class-Specific Patterns through Efficient Neural Search [43.454121220860564]
  We propose a novel, inherently interpretable binary neural network architecture DNAPS that extracts differential patterns from data. DiffNaps is scalable to hundreds of thousands of features and robust to noise. We show on synthetic and real world data, including three biological applications, that, unlike its competitors, DiffNaps consistently yields accurate, succinct, and interpretable class descriptions.
  arXiv  Detail & Related papers  (2023-12-07T14:09:18Z)
• Information theoretic study of the neural geometry induced by category learning [0.0]
  We take an information theoretic approach to assess the efficiency of the representations induced by category learning. One main consequence is that category learning induces an expansion of neural space near decision boundaries.
  arXiv  Detail & Related papers  (2023-11-27T10:16:22Z)
• Deep Intrinsic Decomposition with Adversarial Learning for Hyperspectral Image Classification [9.051982753583232]
  This work develops a novel deep intrinsic decomposition with adversarial learning, namely AdverDecom, for hyperspectral image classification. A discriminative network is constructed to distinguish different environmental categories. Experiments are conducted over three commonly used real-world datasets.
  arXiv  Detail & Related papers  (2023-10-28T00:41:25Z)
• SimNP: Learning Self-Similarity Priors Between Neural Points [52.4201466988562]
  SimNP is a method to learn category-level self-similarities. We show that SimNP is able to outperform previous methods in reconstructing symmetric unseen object regions.
  arXiv  Detail & Related papers  (2023-09-07T16:02:40Z)
• Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognition [71.35205097460124]
  How humans understand and recognize the actions of others is a complex neuroscientific problem. LA-GCN proposes a graph convolution network using large-scale language models (LLM) knowledge assistance.
  arXiv  Detail & Related papers  (2023-05-21T08:29:16Z)
• Rank Diminishing in Deep Neural Networks [71.03777954670323]
  Rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear.
  arXiv  Detail & Related papers  (2022-06-13T12:03:32Z)
• Encoding Hierarchical Information in Neural Networks helps in Subpopulation Shift [8.01009207457926]
  Deep neural networks have proven to be adept in image classification tasks, often surpassing humans in terms of accuracy. In this work, we study the aforementioned problems through the lens of a novel conditional supervised training framework. We show that learning in this structured hierarchical manner results in networks that are more robust against subpopulation shifts.
  arXiv  Detail & Related papers  (2021-12-20T20:26:26Z)
• Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders [72.15087604017441]
  Category-selectivity describes the observation that certain spatially localized areas of the cerebral cortex tend to respond robustly and selectively to stimuli from specific limited categories. We leverage the newly introduced Topographic Variational Autoencoder to model of the emergence of such localized category-selectivity in an unsupervised manner. We show preliminary results suggesting that our model yields a nested spatial hierarchy of increasingly abstract categories, analogous to observations from the human ventral temporal cortex.
  arXiv  Detail & Related papers  (2021-10-25T11:37:41Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View [82.80085730891126]
  We provide the first modernally precise analysis of linear multiclass classification. Our analysis reveals that the classification accuracy is highly distribution-dependent. The insights gained may pave the way for a precise understanding of other classification algorithms.
  arXiv  Detail & Related papers  (2020-11-16T05:17:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.