On Subquadratic Architectures: From Applications to Principles

• URL: http://arxiv.org/abs/2606.12364v1
• Date: Wed, 10 Jun 2026 17:33:55 GMT
• Title: On Subquadratic Architectures: From Applications to Principles
• Abstract summary: Subquadratic architectures offer a scalable alternative to Transformers.<n>We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet.<n>Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.
• Score: 22.49059700270806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.

Related papers

• Unlocking Feature Learning in Gated Delta Networks at Scale [52.705562336559446]
  Training and scaling Large Language Models demand enormous computational resources.<n>We derive the scaling rules for Gated Delta Network.<n> Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer.
  arXiv  Detail & Related papers  (2026-06-02T08:45:24Z)
• Sketch2Simulation: Automating Flowsheet Generation via Multi Agent Large Language Models [0.0]
  Converting process sketches into executable simulation models remains a major bottleneck in process systems engineering.<n>Recent advances in generative AI have improved both engineering-diagram interpretation and flowsheet generation, but these remain largely disconnected.<n>We present an end-to-end multi-agent large language model system that converts process diagrams directly into executable Aspen HYSYS flowsheets.
  arXiv  Detail & Related papers  (2026-03-25T03:36:46Z)
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity [22.40851170527]
  Scaling laws play a central role in the success of Large Language Models.<n>Recent alternatives such as xLSTM offer linear complexity with respect to context length.<n>xLSTM's advantage widens as training and inference contexts grow.
  arXiv  Detail & Related papers  (2025-10-02T17:14:34Z)
• Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs [51.29260537017623]
  Large Language Models (LLMs) are emerging as versatile foundation models for computational chemistry.<n>These models often lack round-trip consistency.<n>We introduce Round-Trip Reinforcement Learning (RTRL), a novel framework that trains a model to improve its consistency.
  arXiv  Detail & Related papers  (2025-10-01T23:58:58Z)
• A2R: An Asymmetric Two-Stage Reasoning Framework for Parallel Reasoning [57.727084580884075]
  Asymmetric Two-Stage Reasoning framework designed to bridge gap between a model's potential and its actual performance.<n>A2R-Efficient is a "small-to-big" variant that combines a Qwen3-4B explorer with a Qwen3-8B synthesizer.<n>Results show A2R is not only a performance-boosting framework but also an efficient and practical solution for real-world applications.
  arXiv  Detail & Related papers  (2025-09-26T08:27:03Z)
• LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence [61.46575527504109]
  LimiX-16M and LimiX-2M treat structured data as a joint distribution over variables and missingness.<n>We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios.
  arXiv  Detail & Related papers  (2025-09-03T17:39:08Z)
• StoxLSTM: A Stochastic Extended Long Short-Term Memory Network for Time Series Forecasting [20.120876019697445]
  Extended Long Short-Term Memory (xLSTM) network has attracted widespread research interest due to its enhanced capability to model complex temporal dependencies in diverse time series applications.<n>We propose a xLSTM, termed StoxLSTM, that improves the original architecture into a state space modeling framework by incorporating latent variables within xLSTM.<n>Experiments on publicly available benchmark datasets from multiple research communities demonstrate that StoxLSTM consistently outperforms state-of-the-art baselines with better robustness and stronger generalization ability.
  arXiv  Detail & Related papers  (2025-09-01T07:11:05Z)
• xLSTMAD: A Powerful xLSTM-based Method for Anomaly Detection [0.794682109939797]
  We propose xLSTMAD, the first anomaly detection method that integrates a full encoder-decoder xLSTM architecture.<n>We evaluate our method on the comprehensive TSB-AD-M benchmark, which spans 17 real-world datasets.<n>In our results, xLSTM showcases state-of-the-art accuracy, outperforming 23 popular anomaly detection baselines.
  arXiv  Detail & Related papers  (2025-06-28T10:39:09Z)
• A Deep Learning Framework for Sequence Mining with Bidirectional LSTM and Multi-Scale Attention [11.999319439383918]
  This paper addresses the challenges of mining latent patterns and modeling contextual dependencies in complex sequence data.<n>A sequence pattern mining algorithm is proposed by integrating Bidirectional Long Short-Term Memory (BiLSTM) with a multi-scale attention mechanism.<n>BiLSTM captures both forward and backward dependencies in sequences, enhancing the model's ability to perceive global contextual structures.
  arXiv  Detail & Related papers  (2025-04-21T16:53:02Z)
• Longhorn: State Space Models are Amortized Online Learners [51.10124201221601]
  State-space models (SSMs) offer linear decoding efficiency while maintaining parallelism during training. In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. We introduce a novel deep SSM architecture, Longhorn, whose update resembles the closed-form solution for solving the online associative recall problem.
  arXiv  Detail & Related papers  (2024-07-19T11:12:08Z)
• FAENet: Frame Averaging Equivariant GNN for Materials Modeling [123.19473575281357]
  We introduce a flexible framework relying on frameaveraging (SFA) to make any model E(3)-equivariant or invariant through data transformations. We prove the validity of our method theoretically and empirically demonstrate its superior accuracy and computational scalability in materials modeling.
  arXiv  Detail & Related papers  (2023-04-28T21:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.