The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling

• URL: http://arxiv.org/abs/2603.07461v1
• Date: Sun, 08 Mar 2026 04:50:29 GMT
• Title: The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling
• Abstract summary: We introduce the Dual-Stream Transformer, which decomposes the residual stream into two functionally distinct components.<n>We measure this tradeoff on language modeling tasks at 29M parameters.
• Score: 1.2891210250935148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard transformers entangle all computation in a single residual stream, obscuring which components perform which functions. We introduce the Dual-Stream Transformer, which decomposes the residual stream into two functionally distinct components: a token stream updated by attention and a context stream updated by feed-forward networks. Information flow between attention heads is controlled through a hierarchy of mixing strategies, from fully independent (maximum interpretability) to dense (standard transformer behavior). This design exposes a tunable tradeoff between interpretability and performance. We measure this tradeoff on language modeling tasks at 29M parameters. Fully independent head mixing increases validation loss by 8\% relative to dense baselines. The recommended Kronecker mixing strategy, which permits scalar communication between heads while preserving within-head structure, costs only 2.5\%. All configurations maintain functional generation under attention amplification (scaling logits by factors up to 16 at inference time), with degradation ranging from 16\% to 27\%. This robustness suggests the architectures learn discrete algorithms that operate independently of soft probabilistic mixing. The architecture provides a foundation for interpretable language models where internal structure is exposed by design. \footnote{This work was partially supported by DARPA Contract HR001125C0302.}

Related papers

• Tracing distinguishability through transformer processing with stochastic LayerNorm [0.0]
  We give representations volume, turning similarity into statistical distinguishability.<n>We realize this idea in pretrained transformers through a light-touch modification to LayerNorm.<n>These results establish distinguishability as a functionally grounded lens on transformer computation.
  arXiv  Detail & Related papers  (2026-08-31T12:57:38Z)
• Unifying Graph Neural Networks Through a Common Layer Equation [75.88127999406392]
  We introduce a common layer equation that represents covered architectures through seven components.<n>The central factorization separates where information moves, encoded by the propagation bank, from what moves, encoded by the message maps.<n>We make this unification explicit and checkable through worked reductions of canonical layers and component assignments.
  arXiv  Detail & Related papers  (2026-08-17T04:37:59Z)
• Scaling Interpretable Transformers with Parity Bottleneck Layers [0.0]
  We introduce a GPT-2-scale architecture whose intermediate representations are efficient and wide / sparse by design.<n>A Deep Parity Bottleneck (DPB) replaces a learned over-complete basis with a parameter-free algebraic dictionary.<n>The ParityTransformer's features are native to the model's forwards pass by construction, addressing the question of whether SAEs probe features the model actually uses during computation.
  arXiv  Detail & Related papers  (2026-07-22T18:25:16Z)
• Variable-Width Transformers [57.07167443557886]
  We investigate nonuniform capacity allocation across network depth by proposing a $times$-shaped > former architecture.<n>Our design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism.<n>By reducing the average layer width, this architecture also requires fewer overall FLOPs and smaller KV cache memory and I/O cost.
  arXiv  Detail & Related papers  (2026-06-16T17:59:03Z)
• Interpretable-by-Design Transformers via Architectural Stream Independence [1.2891210250935148]
  We investigate whether architectural constraints can enforce interpretability by design through architectural stream independence.<n>We validate this principle through the Late Fusion Architecture (LFA), which demonstrates interpretable symbolic heads through all the final layers.<n>We quantify this effect by introducing the Token-Position Dependence Score (PDS), with $PDS_max$ = 0.276 and 0.058 respectively.
  arXiv  Detail & Related papers  (2026-03-08T05:44:18Z)
• Hybrid Dual-Path Linear Transformations for Efficient Transformer Architectures [0.0]
  We introduce the Hybrid Dual-Path Linear (HDPL) operator, which decomposes the affine transformation into two topologically distinct pathways.<n> Experiments on the FineWeb-Edu dataset demonstrate that the HDPL architecture outperforms a standard Llama-style baseline.<n>We discuss how the explicit materialization of a probabilistic latent space within the Transformer backbone serves as a vital architectural affordance.
  arXiv  Detail & Related papers  (2026-02-05T20:16:10Z)
• TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognition [63.93802691275012]
  We propose a lightweight Dual Dynamic Token Mixer (D-Mixer) to simultaneously learn global and local dynamics.<n>We use D-Mixer as the basic building block to design TransXNet, a novel hybrid CNN-Transformer vision backbone network.<n>In the ImageNet-1K classification, TransXNet-T surpasses Swin-T by 0.3% in top-1 accuracy while requiring less than half of the computational cost.
  arXiv  Detail & Related papers  (2023-10-30T09:35:56Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures [57.46093180685175]
  We demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. We add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales.
  arXiv  Detail & Related papers  (2021-06-10T15:41:53Z)
• X-volution: On the unification of convolution and self-attention [52.80459687846842]
  We propose a multi-branch elementary module composed of both convolution and self-attention operation. The proposed X-volution achieves highly competitive visual understanding improvements.
  arXiv  Detail & Related papers  (2021-06-04T04:32:02Z)
• Cascaded Head-colliding Attention [28.293881246428377]
  Transformers have advanced the field of natural language processing (NLP) on a variety of important tasks. We present cascaded head-colliding attention (CODA) which explicitly models the interactions between attention heads through a hierarchical variational distribution.
  arXiv  Detail & Related papers  (2021-05-31T10:06:42Z)
• Reciprocal Adversarial Learning via Characteristic Functions [12.961770002117142]
  Generative adversarial nets (GANs) have become a preferred tool for tasks involving complicated distributions. We show how to use the characteristic function (CF) to compare the distributions rather than their moments. We then prove an equivalence between the embedded and data domains when a reciprocal exists, where we naturally develop the GAN in an auto-encoder structure. This efficient structure uses only two modules, together with a simple training strategy, to achieve bi-directionally generating clear images.
  arXiv  Detail & Related papers  (2020-06-15T14:04:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.