HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models

• URL: http://arxiv.org/abs/2604.22442v1
• Date: Fri, 24 Apr 2026 10:59:30 GMT
• Title: HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
• Abstract summary: We introduce Hub Code, a pluggable module that replaces O(n2) attention layers with O(nM) hub-mediated routing.<n>We demonstrate it in two from-scratch architectures: a Jamba-style hybrid and a 12-layer Transformer.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce HubRouter, a pluggable module that replaces O(n^2) attention layers with O(nM) hub-mediated routing, where M << n is a small number of learned hub tokens. We demonstrate it in two from-scratch architectures: a Jamba-style hybrid and a 12-layer Transformer; retrofit into pretrained models is a tested negative case. HubRouter implements an encode-decode-score-council pipeline: M learned hubs cross-attend to all tokens, tokens project against hubs for routing fingerprints, a score head selects top-k tokens, and a sparse council attends only to the selected subset. We validate HubRouter in three settings. (1) Hub-Jamba yields a nominal 4.2% PPL improvement (200.2 vs 209.0, single seed; possibly within seed noise) and up to ~90x training throughput at sequence length 1024 in matched PyTorch-native baselines; an optimised baseline would narrow this to ~10-15x. (2) Graduated replacement of 25% of Transformer attention layers gives the best perplexity in our matched-budget sweep (268.0 vs 282.4 pure Transformer). (3) Hub-GPT provides strictly causal routing, achieving PPL 211.5 +/- 0.4 over 3 seeds (post council-causal fix); approximately 3 PPL worse than Jamba's 208.5 +/- 0.7, a measurable quality cost for avoiding O(n^2) computation. Post-fix, chunk size C has little effect; the pre-fix chunk-size benefit was an artifact of a bidirectional-council leak we found in adversarial review. A multi-seed hub-count sweep (~105 runs across M=1-32) reveals M=8-14 as the reliably-converging sub-band (4-5/5 seeds); M=6 is rescued to 5/5 by orthogonal regularization, while M>=20 shows increasing seed sensitivity. Companion paper arXiv:2603.20997 (Basu, 2026) defines the routing diagnostic task. Code and scripts will be released.

Related papers

• Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study [0.0]
  We quantify bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB)<n>We build llamamoe-trace, a zero-surgery router-telemetry tool, and measure routing on Q3-30B.<n>We pre-register training of 137M MoE language models with auxiliary locality and domain router losses.
  arXiv  Detail & Related papers  (2026-08-18T19:23:32Z)
• BiKAN: Restoring Collapsed Basis of Binary Kolmogorov--Arnold Networks [6.860988566886594]
  Binarizing a Kolmogorov--Arnold Network (KAN) changes function space available to each layer.<n>Our proposed BiKAN addresses this critical issue by augmenting each binary KAN with selected degree-2 Walsh characters.<n>At W1A1, BiKAN reaches $99.48%$, $84.38%$, and $55.81%$ on M, 401AR-10, and 72AR-100, respectively.
  arXiv  Detail & Related papers  (2026-08-02T20:49:59Z)
• Representation Distribution Matching for One-Step Visual Generation [107.84208192304455]
  We elucidate the design space of Representation Distribution Matching (RDM)<n>We identify two design axes, how the distributions are compared and the representations they are compared in, and controlled studies along them yield three findings.<n>The same recipe post-trains the four-step FLUX.2 [klein] into a one-step generator, surpassing the four-step version on GenEval, 0.826 to 0.794, and on PickScore, 22.76 to 22.58, in 90 H200 GPU-hours.
  arXiv  Detail & Related papers  (2026-07-02T16:15:38Z)
• LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models [0.0]
  We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis.<n> LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, 1.08M parameters) on the Q/K/V/O attention projections.<n>A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term
  arXiv  Detail & Related papers  (2026-06-01T07:50:05Z)
• Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory [1.081571058570587]
  AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism.<n>We introduce DODOCO to test two assumptions about the workload.<n> scaling EP changes the per-expert max/mean token ratio by at most 5% within every architecture's measurable range.
  arXiv  Detail & Related papers  (2026-05-20T10:14:00Z)
• Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality [0.0]
  We build a sparse Mixture-of-Experts (MoE) architecture using cosine-similarity routing against learned centroids.<n>We find that routing topology does not determine perplexity (PPL)<n>Expert-level and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
  arXiv  Detail & Related papers  (2026-04-15T21:02:02Z)
• THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture [0.0]
  THEIA is a 2.75M modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives.<n> Transformer baselines also reach >99% on all 39 rules, and flats match THEIA on Phase-1 accuracy within 0.04pp.
  arXiv  Detail & Related papers  (2026-04-13T10:44:15Z)
• AXELRAM: Quantize Once, Never Dequantize [0.0]
  AXELRAM computes attention scores directly from quantized KV cache indices without dequantization.<n>We trace the root cause to layer-wise norm and propose a gradient-free sign pattern selection (200 candidates, 8 calibration samples, one-time) that eliminates catastrophic spikes with zero additional hardware cost.
  arXiv  Detail & Related papers  (2026-04-03T02:03:38Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• A Simple Linear Patch Revives Layer-Pruned Large Language Models [58.056251480151104]
  Layer pruning has emerged as a widely used technique for compressing large language models (LLMs)<n>textscLinearPatch fuses two operations into one matrix multiply at the pruning interface.<n>The patch can be further refined with 5K unlabeled samples via memory-efficient offline distillation, pushing the retention to 95.16% within only 30 minutes on a single GPU.
  arXiv  Detail & Related papers  (2025-05-30T15:06:08Z)
• EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm [111.17100512647619]
  This paper explains the rationality of Vision Transformer by analogy with the proven practical evolutionary algorithm (EA) We propose a novel pyramid EATFormer backbone that only contains the proposed EA-based transformer (EAT) block. Massive quantitative and quantitative experiments on image classification, downstream tasks, and explanatory experiments demonstrate the effectiveness and superiority of our approach.
  arXiv  Detail & Related papers  (2022-06-19T04:49:35Z)
• Neighborhood Attention Transformer [24.478343522484604]
  We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels. We develop NATTENNeighborhood Attention Extension, which allows NA to run up to 40% faster than Swin's WSA.
  arXiv  Detail & Related papers  (2022-04-14T17:55:15Z)
• DeepNet: Scaling Transformers to 1,000 Layers [106.33669415337135]
  We introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer. In-depth theoretical analysis shows that model updates can be bounded in a stable way. We successfully scale Transformers up to 1,000 layers without difficulty, which is one order of magnitude deeper than previous deep Transformers.
  arXiv  Detail & Related papers  (2022-03-01T15:36:38Z)
• What's Hidden in a One-layer Randomly Weighted Transformer? [100.98342094831334]
  Hidden within one-layer randomly weighted neural networks, there existworks that can achieve impressive performance. Using a fixed pre-trained embedding layer, the previously foundworks are smaller than, but can match 98%/92% (34.14/25.24 BLEU) of the performance of, a trained Transformer small/base on IWSLT14/WMT14.
  arXiv  Detail & Related papers  (2021-09-08T21:22:52Z)
• 1$\times$N Block Pattern for Network Sparsity [90.43191747596491]
  We propose one novel concept of $1times N$ block sparsity pattern (block pruning) to break this limitation. Our pattern obtains about 3.0% improvements over filter pruning in the top-1 accuracy of MobileNet-V2. It also obtains 56.04ms inference savings on Cortex-A7 CPU over weight pruning.
  arXiv  Detail & Related papers  (2021-05-31T05:50:33Z)
• Capsules with Inverted Dot-Product Attention Routing [84.89818784286953]
  We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. Our method improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.
  arXiv  Detail & Related papers  (2020-02-12T02:09:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.