TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal Language Modeling

• URL: http://arxiv.org/abs/2608.22117v1
• Date: Sat, 22 Aug 2026 22:04:39 GMT
• Title: TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal Language Modeling
• Abstract summary: We introduce the TANGO model (Token-Aggregated Gating Operators)<n>Each source token produces a SwiGLU vector gate.<n>We compare TANGO with Recurrent and Untied Transformer++, full-attention GAU, and FLASH.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A standard Transformer block separates cross-token interaction in self-attention from a nonlinear feed-forward network applied independently at each position. We introduce the TANGO model (Token-Aggregated Nonlinear Gating Operators), which replaces these two sublayers with one cross-token gated residual update. Each source token produces a SwiGLU gate vector. Query-key similarities determine a weighted average of source gates for each destination, and the resulting gate rescales projected destination features. TANGO assigns a separate weight to every causally visible source and is quadratic in sequence length. The WANGO model (Windowed Aggregation of Nonlinear Gating Operators) retains the same unnormalized scores within a recent window and uses positive feature-map prefix statistics for older sources, giving linear sequence-length complexity for fixed window and feature dimensions. We compare TANGO and WANGO with Recurrent and Untied Transformer++, full-attention GAU, and FLASH. All models have approximately 44.3M nonembedding parameters and are trained in three matched runs. TANGO, WANGO, and Recurrent Transformer++ apply one shared block four times; the other architectures use four independent blocks. TANGO obtains the lowest mean validation negative log-likelihood on FineWeb-Edu, Lean, and DeepMind Mathematics, although it has the largest analytical forward-pass operation count. WANGO obtains the lowest mean FineWeb-Edu NLL among the architectures with computation linear in sequence length and outperforms Recurrent Transformer++ at nearly the same analytical forward-pass multiply-accumulate count.

Related papers

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models [75.80275843320511]
  We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
  arXiv  Detail & Related papers  (2026-07-09T07:37:59Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• A Dual-Path Architecture for Scaling Compute and Capacity in LLMs [1.7347918181145063]
  We propose a novel dual-path block that can flexibly scale compute, the number of sequential operations applied to a hidden state, and capacity.<n>We show that across two FLOP budgets, our dual-path model surpasses iso-FLOP matched models on language modeling and downstream evaluations.
  arXiv  Detail & Related papers  (2026-05-28T16:41:36Z)
• Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers [32.743518029177146]
  We propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines.<n>Our method decomposes these nonlinear computations into three recurring primitives -- division, exponentiation, and $ell$ norms -- and realizes them via population groups.<n>Experiments show that selectively replacing the targeted nonlinear operators incurs less than a $1% accuracy drop across all evaluated tasks.
  arXiv  Detail & Related papers  (2026-05-19T06:59:46Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling [0.0]
  Gated Associative Memory (GAM) network is a novel, fully parallel architecture for sequence modeling.<n>We implement GAM from scratch and conduct a rigorous comparative analysis against a standard Transformer model and a modern linear-time baseline.<n>Our experiments demonstrate that GAM is consistently faster, outperforming both baselines on training speed, and achieves a superior or competitive final validation perplexity across all datasets.
  arXiv  Detail & Related papers  (2025-08-30T20:59:46Z)
• Comba: Improving Bilinear RNNs with Closed-loop Control [57.800320390698516]
  We introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models.<n>We propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections.<n>We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus.
  arXiv  Detail & Related papers  (2025-06-03T05:44:50Z)
• OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain [24.24834151329251]
  OLinear is a $mathbfo$rthogonally transformed domain that operates in a $mathbfo$rthogonally transformed domain.<n>We introduce a customized linear layer, $mathbfNormLin$, which employs a normalized weight matrix to capture multivariate dependencies.<n>Experiments on 24 benchmarks and 140 forecasting tasks demonstrate that OLinear consistently achieves state-of-the-art performance with high efficiency.
  arXiv  Detail & Related papers  (2025-05-12T10:39:37Z)
• Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator [36.66799002929583]
  The transformer model is known to be computationally demanding, and prohibitively costly for long sequences.<n>We propose Fourier Transformer, a simple yet effective approach by progressively removing redundancies in hidden sequence.<n>Our model achieves state-of-the-art performances among all transformer-based models on the long-range modeling benchmark LRA.
  arXiv  Detail & Related papers  (2023-05-24T12:33:06Z)
• Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences [52.6022911513076]
  Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. We propose Linformer and Informer to reduce the quadratic complexity to linear (modulo logarithmic factors) via low-dimensional projection and row selection. Based on the theoretical analysis, we propose Skeinformer to accelerate self-attention and further improve the accuracy of matrix approximation to self-attention.
  arXiv  Detail & Related papers  (2021-12-10T06:58:05Z)
• Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization [59.87663954467815]
  Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer. In this paper, leveraging a recent linear algebraic theorem called Gershgorin disc perfect alignment (GDPA), we unroll a projection-free algorithm for semi-definite programming relaxation (SDR) of a binary graph. Experimental results show that our unrolled network outperformed pure model-based graph classifiers, and achieved comparable performance to pure data-driven networks but using far fewer parameters.
  arXiv  Detail & Related papers  (2021-09-10T07:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.