M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

• URL: http://arxiv.org/abs/2603.14360v1
• Date: Sun, 15 Mar 2026 12:53:09 GMT
• Title: M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
• Abstract summary: We introduce Matrix-to-Matrix RNN: an architecture with matrix-valued hidden states and expressive non-linear state transitions.<n>We demonstrate that the language modeling performance of non-linear RNNs is limited by their state size.<n>We also demonstrate how the state size expansion mechanism enables efficient use of tensor cores.
• Score: 53.351103753419046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers are highly parallel but are limited to computations in the TC$^0$ complexity class, excluding tasks such as entity tracking and code execution that provably require greater expressive power. Motivated by this limitation, we revisit non-linear Recurrent Neural Networks (RNNs) for language modeling and introduce Matrix-to-Matrix RNN (M$^2$RNN): an architecture with matrix-valued hidden states and expressive non-linear state transitions. We demonstrate that the language modeling performance of non-linear RNNs is limited by their state size. We also demonstrate how the state size expansion mechanism enables efficient use of tensor cores. Empirically, M$^2$RNN achieves perfect state tracking generalization at sequence lengths not seen during training. These benefits also translate to large-scale language modeling. In hybrid settings that interleave recurrent layers with attention, Hybrid M$^2$RNN outperforms equivalent Gated DeltaNet hybrids by $0.4$-$0.5$ perplexity points on a 7B MoE model, while using $3\times$ smaller state sizes for the recurrent layers. Notably, replacing even a single recurrent layer with M$^2$RNN in an existing hybrid architecture yields accuracy gains comparable to Hybrid M$^2$RNN with minimal impact on training throughput. Further, the Hybrid Gated DeltaNet models with a single M$^2$RNN layer also achieve superior long-context generalization, outperforming state-of-the-art hybrid linear attention architectures by up to $8$ points on LongBench. Together, these results establish non-linear RNN layers as a compelling building block for efficient and scalable language models.

Related papers

• Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware [5.066906328188419]
  Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost.<n>We introduce a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss.
  arXiv  Detail & Related papers  (2026-08-31T08:28:01Z)
• Tensor-Augmented Convolutional Neural Networks: Enhancing Expressivity with Generic Tensor Kernels [0.0]
  Convolutional Neural Networks (CNNs) excel at extracting local features hierarchically, but their performance in capturing complex correlations hinges heavily on deep architectures.<n>We propose a physically-guided shallow model: tensor-augmented CNN (TACNN)<n>In our design the convolution output of each layer becomes a multilinear form capable of capturing high-order feature correlations.
  arXiv  Detail & Related papers  (2026-04-09T10:44:23Z)
• Evolution Strategies at the Hyperscale [57.75314521465674]
  We introduce EGGROLL, an evolution strategies (ES) algorithm designed to scale backprop-free optimization to large population sizes.<n>ES is a set of powerful blackbox optimisation methods that can handle non-differentiable or noisy objectives.<n>EGGROLL overcomes these bottlenecks by generating random matrices $Ain mathbbRmtimes r, Bin mathbbRntimes r$ with $rll min(m,n)$ to form a low-rank matrix perturbation $A Btop$
  arXiv  Detail & Related papers  (2025-11-20T18:56:05Z)
• MesaNet: Sequence Modeling by Locally Optimal Test-Time Training [67.45211108321203]
  We introduce a numerically stable, chunkwise parallelizable version of the recently proposed Mesa layer.<n>We show that optimal test-time training enables reaching lower language modeling perplexity and higher downstream benchmark performance than previous RNNs.
  arXiv  Detail & Related papers  (2025-06-05T16:50:23Z)
• Fixed-Point RNNs: Interpolating from Diagonal to Dense [18.06917701940596]
  Linear recurrent neural networks (RNNs) and state-space models (SSMs) have become promising alternatives to softmax-attention as sequence mixing layers in Transformer architectures.<n>Current models, however, do not exhibit the full state-tracking expressivity of RNNs because they rely on channel-wise (i.e. diagonal) sequence mixing.<n>In this paper, we investigate parameterizations of a large class of dense linear RNNs as fixed-points of parallelizable diagonal RNNs.
  arXiv  Detail & Related papers  (2025-03-13T18:50:22Z)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products [60.72655477351486]
  Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling.<n>Existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices.
  arXiv  Detail & Related papers  (2025-02-14T16:59:05Z)
• Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues [65.41946981594567]
  Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers for long sequences.<n>However, both Transformers and LRNNs struggle to perform state-tracking, which may impair performance in tasks such as code evaluation.<n>We show that extending the eigenvalue range of Mamba and DeltaNet to include negative values can improve their performance on state-tracking tasks.
  arXiv  Detail & Related papers  (2024-11-19T14:35:38Z)
• Hierarchically Gated Recurrent Neural Network for Sequence Modeling [36.14544998133578]
  We propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN) Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model.
  arXiv  Detail & Related papers  (2023-11-08T16:50:05Z)
• On the Computational Complexity and Formal Hierarchy of Second Order Recurrent Neural Networks [59.85314067235965]
  We extend the theoretical foundation for the $2nd$-order recurrent network ($2nd$ RNN) We prove there exists a class of a $2nd$ RNN that is Turing-complete with bounded time. We also demonstrate that $2$nd order RNNs, without memory, outperform modern-day models such as vanilla RNNs and gated recurrent units in recognizing regular grammars.
  arXiv  Detail & Related papers  (2023-09-26T06:06:47Z)
• State space models can express n-gram languages [51.823427608117626]
  We build state space language models that can solve the next-word prediction task for languages generated from n-gram rules.<n>Our proof shows how SSMs can encode n-gram rules using new theoretical results on their capacity.<n>We conduct experiments with a small dataset generated from n-gram rules to show how our framework can be applied to SSMs and RNNs obtained through gradient-based optimization.
  arXiv  Detail & Related papers  (2023-06-20T10:41:23Z)
• Asymmetric Certified Robustness via Feature-Convex Neural Networks [11.605936648692543]
  We show that an ICNN can be generalizationd to an adversarial network. Experiments show that the network is far more efficient than any competitive baseline.
  arXiv  Detail & Related papers  (2023-02-03T19:17:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.