DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)

• URL: http://arxiv.org/abs/2604.01880v1
• Date: Thu, 02 Apr 2026 10:39:06 GMT
• Title: DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)
• Abstract summary: This paper introduces DDCL-INCRT, an architecture that determines its own structure during training.<n>Main theoretical finding is that these two mechanisms reinforce each other.<n>At convergence, the network self-organises into a hierarchy of heads ordered by representational granularity.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern neural networks of the transformer family require the practitioner to decide, before training begins, how many attention heads to use, how deep the network should be, and how wide each component should be. These decisions are made without knowledge of the task, producing architectures that are systematically larger than necessary: empirical studies find that a substantial fraction of heads and layers can be removed after training without performance loss. This paper introduces DDCL-INCRT, an architecture that determines its own structure during training. Two complementary ideas are combined. The first, DDCL (Deep Dual Competitive Learning), replaces the feedforward block with a dictionary of learned prototype vectors representing the most informative directions in the data. The prototypes spread apart automatically, driven by the training objective, without explicit regularisation. The second, INCRT (Incremental Transformer), controls the number of heads: starting from one, it adds a new head only when the directional information uncaptured by existing heads exceeds a threshold. The main theoretical finding is that these two mechanisms reinforce each other: each new head amplifies prototype separation, which in turn raises the signal triggering the next addition. At convergence, the network self-organises into a hierarchy of heads ordered by representational granularity. This hierarchical structure is proved to be unique and minimal, the smallest architecture sufficient for the task, under the stated conditions. Formal guarantees of stability, convergence, and pruning safety are established throughout. The architecture is not something one designs. It is something one derives.

Related papers

• The Giant Hippocampus: From Structural Monoculture to a System of Systems [0.0]
  AI researchers describe state-of-the-art models as one thing repeated at scale: the Transformer, wired identically for text, pixels, or speech.<n>Neuroscientists describe the cortex as a mosaic - dense Layer 4 in visual cortex for spatial encoding, thick Layers 5/6 in motion cortex for temporal integration.<n>This paper argues the gap is a structural error, not a stylistic one, and is measurable.
  arXiv  Detail & Related papers  (2026-07-22T09:55:25Z)
• INCRT: An Incremental Transformer That Determines Its Own Architecture [0.0]
  INCRT (Incremental Transformer) is an architecture that determines its own structure during training.<n>Each growth decision is driven by a single, online-computable geometric quantity.<n>Experiments on SARS-CoV-2 variant classification and SST-2 sentiment analysis confirm both results.
  arXiv  Detail & Related papers  (2026-04-12T15:56:47Z)
• Bearing Syntactic Fruit with Stack-Augmented Neural Networks [59.49467149799849]
  We show that stack-augmented neural networks are more accurate models of human language acquisition than standard architectures.<n>We also propose a modification to the stack RNN architecture that improves hierarchical generalization.
  arXiv  Detail & Related papers  (2025-11-05T15:30:58Z)
• What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains [64.31313691823088]
  In-context learning (ICL) is a capability of transformers, through which trained models learn to adapt to new tasks by leveraging information from the input context.<n>We show that a two-layer transformer with one head per layer can indeed represent any conditional k-gram.
  arXiv  Detail & Related papers  (2025-08-10T07:03:01Z)
• Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi [0.0]
  Convolutional neural networks (CNNs) evaluate short-range correlations in input images which progress along the layers.<n> vision transformer (ViT) architectures evaluate long-range correlations, using repeated transformer encoders composed of fully connected layers.<n>This study demonstrates that CNNs and ViT architectures stem from a unified underlying learning mechanism.
  arXiv  Detail & Related papers  (2025-01-22T14:19:48Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Structural Credit Assignment with Coordinated Exploration [0.0]
  Methods aimed at improving structural credit assignment can generally be classified into two categories. We propose the use of Boltzmann machines or a recurrent network for coordinated exploration. Experimental results demonstrate that coordinated exploration significantly exceeds independent exploration in training speed.
  arXiv  Detail & Related papers  (2023-07-25T04:55:45Z)
• Dynamic Perceiver for Efficient Visual Recognition [87.08210214417309]
  We propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features.
  arXiv  Detail & Related papers  (2023-06-20T03:00:22Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• End to End Binarized Neural Networks for Text Classification [4.046236197219608]
  We propose an end to end binarized neural network architecture for the intent classification task. The proposed architecture achieves comparable to the state-of-the-art results on standard intent classification datasets.
  arXiv  Detail & Related papers  (2020-10-11T11:21:53Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.