Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm

• URL: http://arxiv.org/abs/2607.16295v1
• Date: Mon, 13 Jul 2026 09:54:35 GMT
• Title: Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm
• Abstract summary: Group-Contrastive Forward-Forward (GCFF) is a forward-forward training algorithm that combines class-specific routing with within-class contrastive objectives.<n>On CLIP representations, a single trained GCFF module recovers monosemantic neurons whose abstraction increases progressively with depth.
• Score: 4.838965998214037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mechanistic interpretability has made significant strides in understanding neural network representations, with sparse dictionary learning (SDL) methods, most prominently sparse autoencoders, as a central paradigm. However, recent work has reported several limitations of this paradigm: SDL objectives are non-identifiable; SDL methods rely heavily on the Linear Representation Hypothesis; and a growing body of evidence points to concepts that are encoded non-linearly and are therefore not expressible as any single direction. We hypothesise that a different route to monosemanticity is available. Biological visual systems exhibit highly selective neurons organised into hierarchies of increasing abstraction, and this organisation emerges from local, layer-wise learning rules rather than from a global error signal; we therefore ask whether a biologically plausible learning algorithm will likewise yield monosemantic neurons. To test this, we propose Group-Contrastive Forward-Forward (GCFF), a forward-forward training algorithm that combines class-specific routing with within-class contrastive objectives, reaching monosemanticity through architectural constraints rather than sparsity. Because GCFF attaches multiple non-linear layers to the representation under study, its neurons can therefore capture the non-linear concepts. On CLIP representations, a single trained GCFF module recovers monosemantic neurons whose abstraction increases progressively with depth, reaching environmental properties that hold independently of an image's foreground, without any sparsity constraint or supervision of abstraction level. We further demonstrate that GCFF can train networks from scratch, achieving state-of-the-art performance among forward-forward algorithms on various image classification benchmarks.

Related papers

• Convergent Evolution in Neural Representation Space: Emergent Order in Deep Belief Networks [2.925652638976278]
  Deep Belief Networks (DBNs) learn hierarchical generative models without class supervision.<n>We analyze successive layers of DBNs trained on MNIST, Fashion-MNIST, and KMNIST.<n>We find that class-specific clustering generally increases with depth across datasets and network widths.
  arXiv  Detail & Related papers  (2026-08-06T13:05:37Z)
• Toward a Multi-View Brain Network Foundation Model: Cross-View Consistency Learning Across Arbitrary Atlases [62.33465338932216]
  MV-BrainFM is a multi-view brain network foundation model designed to learn generalizable and scalable representations from brain networks constructed with arbitrary atlases.<n>Experiments on more than 20K subjects from 17 fMRI datasets show that MV-BrainFM consistently outperforms 14 existing brain network foundation models.
  arXiv  Detail & Related papers  (2026-03-20T11:55:00Z)
• Concept-Guided Interpretability via Neural Chunking [64.6429903327095]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract recurring chunks on a neural population level.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• URLOST: Unsupervised Representation Learning without Stationarity or Topology [26.010647961403148]
  We introduce a novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology.<n>Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder.<n>We evaluate its effectiveness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions.
  arXiv  Detail & Related papers  (2023-10-06T18:00:02Z)
• A Unified Algebraic Perspective on Lipschitz Neural Networks [88.14073994459586]
  This paper introduces a novel perspective unifying various types of 1-Lipschitz neural networks. We show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers.
  arXiv  Detail & Related papers  (2023-03-06T14:31:09Z)
• Bio-Inspired, Task-Free Continual Learning through Activity Regularization [3.5502600490147196]
  Continual learning approaches usually require discrete task boundaries. We take inspiration from neuroscience, where sparse, non-overlapping neuronal representations have been suggested to prevent forgetting. In addition to sparsity, we introduce lateral recurrent connections within each layer to further protect previously learned representations. Our method achieves similar performance to well-known CL methods, such as Elastic Weight Consolidation and Synaptic Intelligence, without requiring information about task boundaries.
  arXiv  Detail & Related papers  (2022-12-08T15:14:20Z)
• Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks [4.153804257347222]
  We present Agglomerator, a framework capable of providing a representation of part-whole hierarchies from visual cues. We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100.
  arXiv  Detail & Related papers  (2022-03-07T10:56:13Z)
• Self-Ensembling GAN for Cross-Domain Semantic Segmentation [107.27377745720243]
  This paper proposes a self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain data for semantic segmentation. In SE-GAN, a teacher network and a student network constitute a self-ensembling model for generating semantic segmentation maps, which together with a discriminator, forms a GAN. Despite its simplicity, we find SE-GAN can significantly boost the performance of adversarial training and enhance the stability of the model.
  arXiv  Detail & Related papers  (2021-12-15T09:50:25Z)
• Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM) [2.1629276606305057]
  A central goal in deep learning is to learn compact representations of features at every layer of a neural network. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM) We show that our proposed DAM approach has remarkably good performance over various applications.
  arXiv  Detail & Related papers  (2021-10-01T23:31:46Z)
• Exploiting Heterogeneity in Operational Neural Networks by Synaptic Plasticity [87.32169414230822]
  Recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) In this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the Synaptic Plasticity paradigm that poses the essential learning theory in biological neurons. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs.
  arXiv  Detail & Related papers  (2020-08-21T19:03:23Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.