N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation

• URL: http://arxiv.org/abs/2605.13190v1
• Date: Wed, 13 May 2026 08:46:17 GMT
• Title: N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
• Abstract summary: N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
• Score: 68.47358899451255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Improving the inference efficiency of autoregressive transformers typically means reducing FLOPs per token, usually through approximations that degrade model quality. We introduce N-vium, a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware, increasing effective FLOPs per second rather than minimizing compute per token. N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits, with token-adaptive routing. This formulation strictly generalizes the standard transformer, which is recovered exactly when routing assigns zero mass to all intermediate heads. Sampling from the mixture is exact, and complete KV caches are recovered by deferring the upper-layer computation and batching it with later tokens. We pretrain N-vium at scales up to 1.5B parameters. Our largest model reaches 57.9% wall-clock speedup over a parameter- and data-matched standard transformer at no perplexity cost.

Related papers

• SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers [50.00050394179417]
  Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size.<n>We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache.<n>Through a series of ablations, we arrive at a recipe we call SMELT, which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets.
  arXiv  Detail & Related papers  (2026-09-01T14:52:57Z)
• Full-bandwidth transformer [19.176565344686043]
  Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.<n>We introduce the emphfull-bandwidth transformer, which widens this channel with emphlatent feedback<n>Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget.
  arXiv  Detail & Related papers  (2026-08-09T19:59:45Z)
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding [48.9323408950142]
  We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations.<n>We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-gressive recurrent updates under mild assumptions.
  arXiv  Detail & Related papers  (2026-04-23T02:12:58Z)
• Adaptive Pruning of Pretrained Transformer via Differential Inclusions [48.47890215458465]
  Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio.<n>We propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter.<n>This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure.
  arXiv  Detail & Related papers  (2025-01-06T06:34:52Z)
• Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series Forecasting [28.646457377816795]
  We introduce the Sparse Vector Quantized FFN-Free Transformer (Sparse-VQ) Our methodology capitalizes on a sparse vector quantization technique coupled with Reverse Instance Normalization (RevIN) to reduce noise impact. Our FFN-free approach trims the parameter count, enhancing computational efficiency and reducing overfitting.
  arXiv  Detail & Related papers  (2024-02-08T17:09:12Z)
• MixFormerV2: Efficient Fully Transformer Tracking [49.07428299165031]
  Transformer-based trackers have achieved strong accuracy on the standard benchmarks. But their efficiency remains an obstacle to practical deployment on both GPU and CPU platforms. We propose a fully transformer tracking framework, coined as emphMixFormerV2, without any dense convolutional operation and complex score prediction module.
  arXiv  Detail & Related papers  (2023-05-25T09:50:54Z)
• Adaptive Split-Fusion Transformer [90.04885335911729]
  We propose an Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Experiments on standard benchmarks, such as ImageNet-1K, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-26T10:00:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.