Olmo Hybrid: From Theory to Practice and Back

• URL: http://arxiv.org/abs/2604.03444v2
• Date: Tue, 07 Apr 2026 06:15:48 GMT
• Title: Olmo Hybrid: From Theory to Practice and Back
• Abstract summary: We show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both.<n>We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations.<n>Our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm.
• Score: 112.70006263561073
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention. Yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. To address this, we provide evidence for the advantages of hybrid models over pure transformers on several fronts. First, theoretically, we show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Putting this theory to practice, we train Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B but with the sliding window layers replaced by Gated DeltaNet layers. We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, demonstrating the benefit of hybrid models in a controlled, large-scale setting. We find that the hybrid model scales significantly more efficiently than the transformer, explaining its higher performance. However, its unclear why greater expressivity on specific formal problems should result in better scaling or superior performance on downstream tasks unrelated to those problems. To explain this apparent gap, we return to theory and argue why increased expressivity should translate to better scaling efficiency, completing the loop. Overall, our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.

Related papers

• Morphing into Hybrid Attention Models [52.02140767770562]
  Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention.<n>Existing hybrid layer selection methods typically rely on fixed placement patterns or layerwise scoring, implicitly treating layer importance as isolated and overlooking interdependent layer effect under a global hybrid configuration.<n>We propose FlashMorph, an effective, efficient and scalable layer selection method for Transformer-to-hybrid conversion.
  arXiv  Detail & Related papers  (2026-06-29T17:02:34Z)
• Priming: Hybrid State Space Models From Pre-trained Transformers [45.220597209553866]
  Priming is a method that turns Hybrid architecture design from a pre-training problem into a knowledge transfer one.<n>We evaluate, Gated KalmaNet (GKA), Gated DeltaNet (GDN), and Mamba-2, and show that their hierarchy, GKA>GDN>Mamba-2, directly predicts downstream performance on long-context reasoning tasks.<n>Our Hybrid GKA 32B improves over its source Qwen3-32B by +3.8 average reasoning points, while staying within 1% of a Transformer post-trained on the same data.
  arXiv  Detail & Related papers  (2026-05-08T11:43:51Z)
• Reasoning Primitives in Hybrid and Non-Hybrid LLMs [10.629439705877054]
  We study reasoning through two primitives, recall and state-tracking.<n>We use matched Olmo3 transformer and hybrid models in instruction-tuned and reasoning-augmented variants.
  arXiv  Detail & Related papers  (2026-04-23T09:13:28Z)
• Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models [50.45915413315706]
  We prove the existence of fundamental limitations for non-hybrid models.<n>We construct hybrid models of small size and working memory that provably solve these tasks.<n>We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.
  arXiv  Detail & Related papers  (2026-03-09T19:20:01Z)
• Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts [27.8245634187787]
  We present HALO, a pipeline for distilling Transformer models into RNN-attention hybrid models.<n>We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme.<n>The conversion requires just 2.3B tokens, less than 0.01% of their pre-training data.
  arXiv  Detail & Related papers  (2026-01-29T18:59:53Z)
• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step.<n>Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation [37.21518386315535]
  Scaling language models to handle longer contexts introduces substantial memory challenges.<n>We propose LightTransfer, a method that transforms models such as LLaMA into hybrid variants.<n>Our approach identifies lazy layers -- those focusing on recent or initial tokens -- and replaces their full attention with streaming attention.
  arXiv  Detail & Related papers  (2024-10-17T17:58:14Z)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models [76.64055251296548]
  We show how to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources.<n>The resulting hybrid model achieves performance comparable to the original Transformer in chat benchmarks.<n>We also introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models.
  arXiv  Detail & Related papers  (2024-08-27T17:56:11Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Robust Hybrid Learning With Expert Augmentation [31.911717646180886]
  We introduce a hybrid data augmentation strategy termed textitexpert augmentation We demonstrate that expert augmentation, which can be incorporated into existing hybrid systems, improves generalization. We also assess the potential real-world applicability of expert augmentation on a dataset of a real double pendulum.
  arXiv  Detail & Related papers  (2022-02-08T14:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.