Where Should LoRA Go? Component-Type Placement in Hybrid Language Models

• URL: http://arxiv.org/abs/2604.22127v1
• Date: Fri, 24 Apr 2026 00:20:29 GMT
• Title: Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
• Abstract summary: We study component-type LoRA placement across two hybrid architectures.<n>We find that the attention pathway consistently outperforms full-model adaptation with 5-10x fewer trainable parameters.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniformly without considering the distinct functional roles of each component type. We systematically study component-type LoRA placement across two hybrid architectures -- Qwen3.5-0.8B (sequential, GatedDeltaNet + softmax attention) and Falcon-H1-0.5B (parallel, Mamba-2 SSM + attention) -- fine-tuned on three domains and evaluated on five benchmarks. We find that the attention pathway -- despite being the minority component -- consistently outperforms full-model adaptation with 5-10x fewer trainable parameters. Crucially, adapting the recurrent backbone is destructive in sequential hybrids (-14.8 pp on GSM8K) but constructive in parallel ones (+8.6 pp). We further document a transfer asymmetry: parallel hybrids exhibit positive cross-task transfer while sequential hybrids suffer catastrophic forgetting. These results establish that hybrid topology fundamentally determines adaptation response, and that component-aware LoRA placement is a necessary design dimension for hybrid architectures.

Related papers

• Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design [19.539663057036375]
  Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains.<n>We seek an evidence-grounded basis in head-level functional organization learned by RoPE-based Transformers.<n>We instantiate them in Head-wise Hybrid Architecture (HwH), using NoPE FA for global retrieval and LA for local positional modeling.
  arXiv  Detail & Related papers  (2026-09-02T14:49:30Z)
• Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers [54.01042826416009]
  Visual generation requires high-resolution images, long videos, and multimodal context.<n>We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe.<n>We show that the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan2.1 2B baseline.
  arXiv  Detail & Related papers  (2026-07-30T17:58:02Z)
• Revisiting Action Factorization for Complex Action Spaces [0.0]
  This article provides a cross-sectional study of factorization methods on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]<n>We find that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albiet both at increased computational cost.
  arXiv  Detail & Related papers  (2026-06-25T03:48:56Z)
• Component-Aware Self-Speculative Decoding in Hybrid Language Models [0.0]
  Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model.<n>We introduce component-aware self-speculative decoding, isolating the SSM/linear-attention subgraph as a zero-cost internal draft.<n>We show that generic LayerSkip achieves 12x higher acceptance rates than the component-aware strategy.
  arXiv  Detail & Related papers  (2026-05-01T21:25:25Z)
• Functional Component Ablation Reveals Specialization Patterns in Hybrid Language Model Architectures [0.0]
  We present a functional component ablation framework applied to two sub-1B hybrid models.<n>We find that both component types are essential and neither is bypassed.<n>These results provide actionable guidance for hybrid model compression, architecture design, and fault-tolerant deployment.
  arXiv  Detail & Related papers  (2026-03-23T18:41:49Z)
• MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting [6.335488846185043]
  MSLoRA is a backbone-agnostic, parameter-efficient adapter that reweights feature responses rather than re-tuning the underlying backbone.<n>MSLoRA unifies adaptation for both convolutional neural networks (CNNs) and vision transformers (ViTs)
  arXiv  Detail & Related papers  (2025-11-16T00:35:37Z)
• Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning [73.10669391954801]
  We present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0.<n>Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention.<n>Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%.
  arXiv  Detail & Related papers  (2025-10-22T07:59:38Z)
• Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks [5.877451898618022]
  FlowHN is a novel parallel hybrid network architecture that accommodates various strategies for load balancing.<n>Two innovative differentiating factors in FlowHN include a FLOP aware dynamic token split between the attention and SSM branches.
  arXiv  Detail & Related papers  (2025-05-26T03:52:22Z)
• HyMamba: Mamba with Hybrid Geometry-Feature Coupling for Efficient Point Cloud Classification [7.139631485661567]
  HyMamba is a geometry and feature coupled Mamba framework featuring: (1) Geometry-Feature Coupled Pooling (GFCP), which dynamically aggregating adjacent geometric information into local features; (2) Collaborative Feature Enhancer (CoFE), which enhances sparse signal capture through cross-path feature hybridization;.<n>The proposed model achieves superior classification performance, particularly on the ModelNet40 dataset, where it elevates accuracy to 95.99% with merely 0.03M additional parameters. Furthermore, it attains 98.9% accuracy on the ModelNetShot dataset, validating its robust generalization capabilities under sparse samples.
  arXiv  Detail & Related papers  (2025-05-16T10:30:20Z)
• Hymba: A Hybrid-head Architecture for Small Language Models [65.94140459055244]
  Hymba is a family of small language models featuring a hybrid-head parallel architecture. We introduce learnable meta tokens that are prepended to prompts, storing critical information. This model is further optimized by incorporating cross-layer key-value sharing and partial sliding window attention.
  arXiv  Detail & Related papers  (2024-11-20T19:51:25Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• Were RNNs All We Needed? [55.822693848969855]
  In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs)<n>We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that use fewer parameters than their traditional counterparts, are fully parallelizable during training, and achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.
  arXiv  Detail & Related papers  (2024-10-02T03:06:49Z)
• Adaptive Split-Fusion Transformer [90.04885335911729]
  We propose an Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Experiments on standard benchmarks, such as ImageNet-1K, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-26T10:00:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.