INCRT: An Incremental Transformer That Determines Its Own Architecture

• URL: http://arxiv.org/abs/2604.10703v1
• Date: Sun, 12 Apr 2026 15:56:47 GMT
• Title: INCRT: An Incremental Transformer That Determines Its Own Architecture
• Abstract summary: INCRT (Incremental Transformer) is an architecture that determines its own structure during training.<n>Each growth decision is driven by a single, online-computable geometric quantity.<n>Experiments on SARS-CoV-2 variant classification and SST-2 sentiment analysis confirm both results.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer architectures are designed by trial and error: the number of attention heads, the depth, and the head size are fixed before training begins, with no mathematical principle to guide the choice. The result is systematic structural redundancy -- between half and four-fifths of all heads in a trained model can be removed without measurable loss -- because the architecture allocates capacity without reference to the actual requirements of the task.This paper introduces INCRT (Incremental Transformer), an architecture that determines its own structure during training. Starting from a single head, INCRT adds one attention head at a time whenever its current configuration is provably insufficient, and prunes heads that have become redundant. Each growth decision is driven by a single, online-computable geometric quantity derived from the task's directional structure, requiring no separate validation phase and no hand-tuned schedule. Two theorems form the theoretical backbone. The first (homeostatic convergence) establishes that the system always reaches a finite stopping configuration that is simultaneously minimal (no redundant heads) and sufficient (no uncaptured directional energy above the threshold). The second (compressed-sensing analogy) provides a geometric upper bound on the number of heads that this configuration can contain, as a function of the spectral complexity of the task. Experiments on SARS-CoV-2 variant classification and SST-2 sentiment analysis confirm both results: the predicted and observed head counts agree within 12% across all benchmarks, and the final architectures match or exceed BERT-base on distribution-specific tasks while using between three and seven times fewer parameters and no pre-training.

Related papers

• The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems [0.0]
  This thesis turns impossibility results from curiosities into design rules.<n>Its flagship result proves an accuracy ceiling set by architecture alone.<n>The same argument recasts across subfields.
  arXiv  Detail & Related papers  (2026-05-21T20:48:35Z)
• DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations) [0.0]
  This paper introduces DDCL-INCRT, an architecture that determines its own structure during training.<n>Main theoretical finding is that these two mechanisms reinforce each other.<n>At convergence, the network self-organises into a hierarchy of heads ordered by representational granularity.
  arXiv  Detail & Related papers  (2026-04-02T10:39:06Z)
• AgenticRS-EnsNAS: Ensemble-Decoupled Self-Evolving Architecture Search [10.111487060179252]
  This work introduces Ensemble-Decoupled Architecture Search.<n>It predicts system-level performance from single-learner evaluation.<n>It reduces per-candidate search cost from O(M) to O(1) while maintaining O(M) deployment cost only for validated winners.
  arXiv  Detail & Related papers  (2026-03-20T14:57:15Z)
• PRISM: Parallel Residual Iterative Sequence Model [52.26239951489612]
  We propose PRISM (Parallel Residual Iterative Sequence Model) to resolve this tension.<n>PRISM introduces a solver-inspired inductive bias that captures key structural properties of multi-step refinement in a parallelizable form.<n>We prove that this formulation achieves Rank-$L$ accumulation, structurally expanding the update manifold beyond the single-step Rank-$1$ bottleneck.
  arXiv  Detail & Related papers  (2026-02-11T12:39:41Z)
• Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems [59.94955550958074]
  We study a tractable theoretical framework in which each transformer block acts as an expert governed by a continuously trained gating network.<n>We show that expert specialization reduces gradient conflicts and makes each subtask strongly convex.<n>We prove that the training drives the expected prediction loss to near zero in $O(log(epsilon-1)$ steps, significantly improving over the $O(epsilon-1)$ rate for a single transformer.
  arXiv  Detail & Related papers  (2025-10-30T21:07:36Z)
• The Effect of Attention Head Count on Transformer Approximation [26.943083432025926]
  We study the approximation properties of transformers, with particular emphasis on the role of the number of attention heads.<n>Specifically, we show that transformers with sufficiently many heads admit efficient approximation, whereas with too few heads, the number of parameters must scale at least as $O(1/epsiloncT)$, for some constant $c$ and sequence length $T$.
  arXiv  Detail & Related papers  (2025-10-08T05:27:25Z)
• What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains [64.31313691823088]
  In-context learning (ICL) is a capability of transformers, through which trained models learn to adapt to new tasks by leveraging information from the input context.<n>We show that a two-layer transformer with one head per layer can indeed represent any conditional k-gram.
  arXiv  Detail & Related papers  (2025-08-10T07:03:01Z)
• ANDHRA Bandersnatch: Training Neural Networks to Predict Parallel Realities [0.0]
  This work introduces a novel neural network architecture that splits the same input signal into parallel branches at each layer.<n>The branched layers do not merge and form separate network paths, leading to multiple network heads for output prediction.
  arXiv  Detail & Related papers  (2024-11-28T15:36:34Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Unified Field Theory for Deep and Recurrent Neural Networks [56.735884560668985]
  We present a unified and systematic derivation of the mean-field theory for both recurrent and deep networks. We find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in $1/n$.
  arXiv  Detail & Related papers  (2021-12-10T15:06:11Z)
• Contextualized Embeddings based Convolutional Neural Networks for Duplicate Question Identification [0.25782420501870296]
  Question Paraphrase Identification (QPI) is a critical task for large-scale Question-Answering forums. We propose a novel architecture combining a Bidirectional Transformer with Convolutional Neural Networks for the QPI task. Experimental results demonstrate that our model achieves state-of-the-art performance on the Quora Question Pairs dataset.
  arXiv  Detail & Related papers  (2021-09-03T14:30:09Z)
• X-volution: On the unification of convolution and self-attention [52.80459687846842]
  We propose a multi-branch elementary module composed of both convolution and self-attention operation. The proposed X-volution achieves highly competitive visual understanding improvements.
  arXiv  Detail & Related papers  (2021-06-04T04:32:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.