Winner-Take-All bottlenecks enforce disentangled symbolic representations in multi-task learning

• URL: http://arxiv.org/abs/2605.22472v1
• Date: Thu, 21 May 2026 13:33:08 GMT
• Title: Winner-Take-All bottlenecks enforce disentangled symbolic representations in multi-task learning
• Abstract summary: Winner-take-all (WTA) networks constitute a central circuit motif in cortical networks of the brain.<n>We show that a WTA bottleneck within a deep neural network can enforce under certain well-defined conditions the extraction of categorical latent factors of the data.
• Score: 6.244816393907943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Winner-take-all (WTA) networks constitute a central circuit motif in cortical networks of the brain. In addition, WTA-like activations are abundant in modern deep learning models in the form of the softmax activation for example in attention layers of transformers. While their role in the extraction of latent factors has been studied for relatively simple generative models, their role in the context of highly non-linearly entangled latent factors has remained elusive. In this article, we show that a WTA bottleneck within a deep neural network can enforce under certain well-defined conditions the extraction of categorical latent factors of the data in a multi-task learning setup. In particular, we prove that the representation that emerges in the WTA bottleneck is highly symbolic, where a single neuron or a population of neurons encodes the presence of a single abstract feature such as a specific object, color, or position. We furthermore show empirically on two datasets, that this also holds for architectures and setups that do not fully comply with the assumptions of our theorem and demonstrate the advantages of the acquired symbolic representation for generalization. Our proposed model provides insights into the generalization capabilities of deep neural networks with WTA-like components and may serve as an interface between symbolic and subsymbolic AI systems.

Related papers

• A mathematical theory for understanding when abstract representations emerge in neural networks [4.415536082342714]
  We show that abstract representations of latent variables are guaranteed to appear in the last hidden layer of feedforward nonlinear networks.<n>These representations reflect the structure of the desired outputs or the semantics of the input stimuli.
  arXiv  Detail & Related papers  (2025-10-10T19:30:57Z)
• Understanding Task Representations in Neural Networks via Bayesian Ablation [1.3980986259786223]
  We introduce a novel probabilistic framework for interpreting latent task representations in neural networks.<n>Inspired by Bayesian inference, our approach defines a distribution over representational units to infer their causal contributions to task performance.
  arXiv  Detail & Related papers  (2025-05-19T21:36:09Z)
• Deriving Equivalent Symbol-Based Decision Models from Feedforward Neural Networks [0.0]
  Despite its rapid adoption, the opacity of AI systems poses significant challenges to trust and acceptance.<n>This work focuses on the derivation of symbolic models, such as decision trees, from feed-forward neural networks (FNNs)
  arXiv  Detail & Related papers  (2025-04-16T19:22:53Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Neural Clustering based Visual Representation Learning [61.72646814537163]
  Clustering is one of the most classic approaches in machine learning and data analysis. We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data. FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
  arXiv  Detail & Related papers  (2024-03-26T06:04:50Z)
• A Relational Inductive Bias for Dimensional Abstraction in Neural Networks [3.5063551678446494]
  This paper investigates the impact of the relational bottleneck on the learning of factorized representations conducive to compositional coding. We demonstrate that such a bottleneck not only improves generalization and learning efficiency, but also aligns network performance with human-like behavioral biases.
  arXiv  Detail & Related papers  (2024-02-28T15:51:05Z)
• Discrete, compositional, and symbolic representations through attractor dynamics [51.20712945239422]
  We introduce a novel neural systems model that integrates attractor dynamics with symbolic representations to model cognitive processes akin to the probabilistic language of thought (PLoT) Our model segments the continuous representational space into discrete basins, with attractor states corresponding to symbolic sequences, that reflect the semanticity and compositionality characteristic of symbolic systems through unsupervised learning, rather than relying on pre-defined primitives. This approach establishes a unified framework that integrates both symbolic and sub-symbolic processing through neural dynamics, a neuroplausible substrate with proven expressivity in AI, offering a more comprehensive model that mirrors the complex duality of cognitive operations
  arXiv  Detail & Related papers  (2023-10-03T05:40:56Z)
• LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning [73.98142349171552]
  LOGICSEG is a holistic visual semantic that integrates neural inductive learning and logic reasoning with both rich data and symbolic knowledge. During fuzzy logic-based continuous relaxation, logical formulae are grounded onto data and neural computational graphs, hence enabling logic-induced network training. These designs together make LOGICSEG a general and compact neural-logic machine that is readily integrated into existing segmentation models.
  arXiv  Detail & Related papers  (2023-09-24T05:43:19Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.