Functional Component Ablation Reveals Specialization Patterns in Hybrid Language Model Architectures

• URL: http://arxiv.org/abs/2603.22473v1
• Date: Mon, 23 Mar 2026 18:41:49 GMT
• Title: Functional Component Ablation Reveals Specialization Patterns in Hybrid Language Model Architectures
• Abstract summary: We present a functional component ablation framework applied to two sub-1B hybrid models.<n>We find that both component types are essential and neither is bypassed.<n>These results provide actionable guidance for hybrid model compression, architecture design, and fault-tolerant deployment.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hybrid language models combining attention with state space models (SSMs) or linear attention offer improved efficiency, but whether both components are genuinely utilized remains unclear. We present a functional component ablation framework applied to two sub-1B hybrid models -- Qwen3.5-0.8B (sequential: Gated DeltaNet + softmax attention) and Falcon-H1-0.5B (parallel: Mamba-2 + attention) -- with a pure Transformer control (Qwen2.5-0.5B). Through group ablations, layer-wise sweeps, positional ablations, matched random controls, and perplexity analysis across five benchmarks, we establish four findings: (1) both component types are essential and neither is bypassed; (2) the alternative component (linear attention or SSM) is the primary language modeling backbone, causing >35,000x perplexity degradation when removed versus ~82x for attention; (3) component importance follows a positional gradient, with early layers being disproportionately critical; and (4) hybrid architectures exhibit 20-119x greater resilience to random layer removal than pure Transformers, revealing built-in functional redundancy between component types. These results provide actionable guidance for hybrid model compression, architecture design, and fault-tolerant deployment.

Related papers

• Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers [54.01042826416009]
  Visual generation requires high-resolution images, long videos, and multimodal context.<n>We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe.<n>We show that the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan2.1 2B baseline.
  arXiv  Detail & Related papers  (2026-07-30T17:58:02Z)
• MambaADv2: Evolving Duality-enhanced State Space Model for Unsupervised Anomaly Detection [108.36437360254605]
  This paper proposes MambaADv2, a framework tailored for multi-class unsupervised anomaly detection.<n>MambaADv2 comprises a pre-trained encoder and a Mamba-inspired decoder, equipped with Duality-enhanced State Space (DSS) modules across multiple scales.<n>The structure of the Hybrid State Space (HSS) block is tailored by following the SSD-based Mamba lineage and incorporating Mamba3-style position-aware state-space modeling.
  arXiv  Detail & Related papers  (2026-06-22T10:14:06Z)
• SUP-MCRL: Subject-aware Unified Pseudo-feature Coded Multimodal Contrastive Representation Learning for EEG Visual Decoding [4.793866271591739]
  Non-invasive brain-computer interfaces exhibit significant performance degradation when moving from controlled laboratory stimuli to real-world natural images.<n>This degradation occurs because multimodal contrastive representation learning models fail to account for semantic consistency and inter-subject variability in neural representation and selective attention.<n>We propose SUP-MCRL, a unified framework integrating three collaborative mechanisms to overcome these limitations.
  arXiv  Detail & Related papers  (2026-06-15T12:06:08Z)
• Component-Aware Self-Speculative Decoding in Hybrid Language Models [0.0]
  Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model.<n>We introduce component-aware self-speculative decoding, isolating the SSM/linear-attention subgraph as a zero-cost internal draft.<n>We show that generic LayerSkip achieves 12x higher acceptance rates than the component-aware strategy.
  arXiv  Detail & Related papers  (2026-05-01T21:25:25Z)
• Where Should LoRA Go? Component-Type Placement in Hybrid Language Models [0.0]
  We study component-type LoRA placement across two hybrid architectures.<n>We find that the attention pathway consistently outperforms full-model adaptation with 5-10x fewer trainable parameters.
  arXiv  Detail & Related papers  (2026-04-24T00:20:29Z)
• HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents [71.09493646932046]
  We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents.<n>The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning.<n>Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro.
  arXiv  Detail & Related papers  (2026-04-08T17:59:48Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation [77.07565723756119]
  We present InternVLA-A1, a vision-language-action model with dynamic predictive capabilities.<n>We pre-train these models on heterogeneous data sources over real-world robot data, synthetic simulation data, and human videos.<n>We evaluate InternVLA-A1 on 12 real-world robotic tasks and a simulation benchmark.
  arXiv  Detail & Related papers  (2026-01-05T18:54:29Z)
• Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning [73.10669391954801]
  We present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0.<n>Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention.<n>Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%.
  arXiv  Detail & Related papers  (2025-10-22T07:59:38Z)
• A Systematic Analysis of Hybrid Linear Attention [11.722015123070957]
  Linear models often suffer from limited recall performance.<n>Our study highlights selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models.<n>Our models are open-sourced at https://huggingface.co/collections/m-hugging-a-p/hybrid-linear-attention-research-686c488a63d609d2f2 0e2b1e.
  arXiv  Detail & Related papers  (2025-07-08T23:54:11Z)
• TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding [19.42637769220759]
  We introduce TransXSSM, a hybrid architecture that coherently integrates the Transformer and SSM layers under this unified positional encoding scheme.<n>At a 4K length, TransXSSM exhibits training and inference speeds that are 42.3% and 29.5% faster, respectively, relative to standard Transformer models.<n>Our results show that unified positional encoding resolves positional incompatibility in hybrid models, enabling efficient, high-performance long-context modeling.
  arXiv  Detail & Related papers  (2025-06-11T08:26:51Z)
• HyMamba: Mamba with Hybrid Geometry-Feature Coupling for Efficient Point Cloud Classification [7.139631485661567]
  HyMamba is a geometry and feature coupled Mamba framework featuring: (1) Geometry-Feature Coupled Pooling (GFCP), which dynamically aggregating adjacent geometric information into local features; (2) Collaborative Feature Enhancer (CoFE), which enhances sparse signal capture through cross-path feature hybridization;.<n>The proposed model achieves superior classification performance, particularly on the ModelNet40 dataset, where it elevates accuracy to 95.99% with merely 0.03M additional parameters. Furthermore, it attains 98.9% accuracy on the ModelNetShot dataset, validating its robust generalization capabilities under sparse samples.
  arXiv  Detail & Related papers  (2025-05-16T10:30:20Z)
• Hymba: A Hybrid-head Architecture for Small Language Models [65.94140459055244]
  Hymba is a family of small language models featuring a hybrid-head parallel architecture. We introduce learnable meta tokens that are prepended to prompts, storing critical information. This model is further optimized by incorporating cross-layer key-value sharing and partial sliding window attention.
  arXiv  Detail & Related papers  (2024-11-20T19:51:25Z)
• MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions [26.862052778354016]
  We propose a single-head gated transformer architecture with convolution-augmented joint self-attentions, named textitMossFormer. MossFormer significantly outperforms the previous models and achieves the state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! benchmarks.
  arXiv  Detail & Related papers  (2023-02-23T07:17:12Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.