Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

• URL: http://arxiv.org/abs/2603.08859v1
• Date: Mon, 09 Mar 2026 19:20:01 GMT
• Title: Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
• Abstract summary: We prove the existence of fundamental limitations for non-hybrid models.<n>We construct hybrid models of small size and working memory that provably solve these tasks.<n>We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.
• Score: 50.45915413315706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hybrid sequence models--combining Transformer and state-space model layers--seek to gain the expressive versatility of attention as well as the computational efficiency of state-space model layers. Despite burgeoning interest in hybrid models, we lack a basic understanding of the settings where--and underlying mechanisms through which--they offer benefits over their constituent models. In this paper, we study this question, focusing on a broad family of core synthetic tasks. For this family of tasks, we prove the existence of fundamental limitations for non-hybrid models. Specifically, any Transformer or state-space model that solves the underlying task requires either a large number of parameters or a large working memory. On the other hand, for two prototypical tasks within this family--namely selective copying and associative recall--we construct hybrid models of small size and working memory that provably solve these tasks, thus achieving the best of both worlds. Our experimental evaluation empirically validates our theoretical findings. Importantly, going beyond the settings in our theoretical analysis, we empirically show that learned--rather than constructed--hybrids outperform non-hybrid models with up to 6x as many parameters. We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.

Related papers

• Reasoning Primitives in Hybrid and Non-Hybrid LLMs [10.629439705877054]
  We study reasoning through two primitives, recall and state-tracking.<n>We use matched Olmo3 transformer and hybrid models in instruction-tuned and reasoning-augmented variants.
  arXiv  Detail & Related papers  (2026-04-23T09:13:28Z)
• Olmo Hybrid: From Theory to Practice and Back [112.70006263561073]
  We show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both.<n>We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations.<n>Our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm.
  arXiv  Detail & Related papers  (2026-04-03T20:36:34Z)
• Modeling Nonlinear Oscillator Networks Using Physics-Informed Hybrid Reservoir Computing [0.0]
  We investigate hybrid reservoir computing, combining reservoir computing with "expert" analytical models.<n>We show that hybrid reservoir computers generally outperform standard reservoir computers and exhibit greater robustness to parameter tuning.<n>There is good performance for dynamical regimes not accessible to the expert model, demonstrating the contribution of the reservoir.
  arXiv  Detail & Related papers  (2024-11-07T15:09:23Z)
• Automatically Learning Hybrid Digital Twins of Dynamical Systems [56.69628749813084]
  Digital Twins (DTs) simulate the states and temporal dynamics of real-world systems. DTs often struggle to generalize to unseen conditions in data-scarce settings. In this paper, we propose an evolutionary algorithm ($textbfHDTwinGen$) to autonomously propose, evaluate, and optimize HDTwins.
  arXiv  Detail & Related papers  (2024-10-31T07:28:22Z)
• FusionBench: A Comprehensive Benchmark of Deep Model Fusion [78.80920533793595]
  Deep model fusion is a technique that unifies the predictions or parameters of several deep neural networks into a single model. FusionBench is the first comprehensive benchmark dedicated to deep model fusion.
  arXiv  Detail & Related papers  (2024-06-05T13:54:28Z)
• Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognition [1.0499611180329804]
  We propose two hybrid quantum-inspired neural networks with adaptive residual and dense connections respectively for pattern recognition.<n>We show the potential superiority of our hybrid models to prevent gradient explosion owing to the sine and cosine functions in the quantum-inspired layers.
  arXiv  Detail & Related papers  (2024-03-09T01:34:26Z)
• Learning Hybrid Interpretable Models: Theory, Taxonomy, and Methods [0.0]
  A hybrid model involves the cooperation of an interpretable model and a complex black box. The advantages of such models over classical ones are two-fold. They grant users precise control over the level of transparency of the system.
  arXiv  Detail & Related papers  (2023-03-08T08:35:29Z)
• Controllable and Compositional Generation with Latent-Space Energy-Based Models [60.87740144816278]
  Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. By composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024x1024.
  arXiv  Detail & Related papers  (2021-10-21T03:31:45Z)
• Hybrid modeling: Applications in real-time diagnosis [64.5040763067757]
  We outline a novel hybrid modeling approach that combines machine learning inspired models and physics-based models. We are using such models for real-time diagnosis applications.
  arXiv  Detail & Related papers  (2020-03-04T00:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.