ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling

• URL: http://arxiv.org/abs/2607.05583v2
• Date: Thu, 09 Jul 2026 02:10:45 GMT
• Title: ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
• Abstract summary: We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative.<n>In a small, 6M matched setting, training and prefill speedups increase with sequence length.<n>We find that in a small, 6M matched setting, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contemporary language models are dominated by the transformer architecture, which leverages self-attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed transformers to effectively model data across a wide range of modalities and contexts. However, transformers, along with their conventional counterparts such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative. ResonatorLM treats token sequences as a single, driven one-dimensional latent field and replaces attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens, and accuracy reaches 61.31 percent (compared to 55.32 percent) on WikiText.

Related papers

• DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams [63.27233749591346]
  Transformer-based models have dramatically increased their size and parameter count to tackle increasingly complex tasks.<n>Stream data inference is typically performed over a sliding temporal window, leading to highly redundant computations.<n>We propose the Deep Continual Transformer (DeepCoT), a redundancy-free encoder-only model that can be applied over existing deep encoder architectures with minimal changes.
  arXiv  Detail & Related papers  (2025-11-21T16:15:43Z)
• Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference [8.527031391688283]
  Kraken is an evolution of the standard Transformer architecture for efficient inference on multi-device systems. When trained on OpenWebText, Kraken models reach a similar perplexity as standard Transformers. When tested on the SuperGLUE benchmark, Kraken speeds up Time To First Token by a mean of 35.6% across a range of model sizes.
  arXiv  Detail & Related papers  (2024-08-14T20:24:03Z)
• Attention as an RNN [66.5420926480473]
  We show that attention can be viewed as a special Recurrent Neural Network (RNN) with the ability to compute its textitmany-to-one RNN output efficiently. We introduce a new efficient method of computing attention's textitmany-to-many RNN output based on the parallel prefix scan algorithm. We show Aarens achieve comparable performance to Transformers on $38$ datasets spread across four popular sequential problem settings.
  arXiv  Detail & Related papers  (2024-05-22T19:45:01Z)
• Repeat After Me: Transformers are Better than State Space Models at Copying [53.47717661441142]
  We show that while generalized state space models are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context.
  arXiv  Detail & Related papers  (2024-02-01T21:44:11Z)
• RWKV: Reinventing RNNs for the Transformer Era [54.716108899349614]
  We propose a novel model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers.
  arXiv  Detail & Related papers  (2023-05-22T13:57:41Z)
• Container: Context Aggregation Network [83.12004501984043]
  Recent finding shows that a simple based solution without any traditional convolutional or Transformer components can produce effective visual representations. We present the model (CONText Ion NERtwok), a general-purpose building block for multi-head context aggregation. In contrast to Transformer-based methods that do not scale well to downstream tasks that rely on larger input image resolutions, our efficient network, named modellight, can be employed in object detection and instance segmentation networks.
  arXiv  Detail & Related papers  (2021-06-02T18:09:11Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)
• Attention is All You Need in Speech Separation [12.57578429586883]
  We propose a novel RNN-free Transformer-based neural network for speech separation. The proposed model achieves state-of-the-art (SOTA) performance on the standard WSJ0-2/3mix datasets.
  arXiv  Detail & Related papers  (2020-10-25T16:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.