cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

• URL: http://arxiv.org/abs/2607.22577v1
• Date: Sat, 06 Jun 2026 13:51:15 GMT
• Title: cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs
• Abstract summary: cMoLLM is a convolutionally gated mixture-of-LLMs that routes over end-to-end streams through fully differentiable dynamic convolution.<n>In GPT-2-style models trained on FineWeb, cMoLLM improves language modeling perplexity and downstream GLUE and SQuAD accuracy under matched compute.
• Score: 3.50654760771471
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes. We aim to scale capacity through MoE-style mixture throughout the LLM pipeline rather than only the FFN. Prior pipeline-level approaches include ParaScale, which introduces virtual tokens and parallel streams but incurs substantial overhead and suffers from homogenized routing and gradient collapse, and AltUp, which uses an auxiliary prediction branch but offers limited adaptivity and slow convergence. We establish that MoE-style mixture layers can be reformulated as variable-kernel dynamic convolutions, where each expert corresponds to a $1{\times}1$ convolutional kernel and routing implements input-conditioned kernel aggregation. Building on this equivalence, we introduce cMoLLM: a convolutionally gated mixture-of-LLMs that routes over end-to-end streams through fully differentiable dynamic convolution. In GPT-2-style models trained on FineWeb, cMoLLM improves language modeling perplexity and downstream GLUE and SQuAD accuracy under matched compute, with better stream utilization, more stable optimization, and favorable scaling compared to ParaScale- and AltUp-style baselines.

Related papers

• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers [42.7047638496247]
  We propose a novel framework for enabling LMC-based model merging to pretrained transformers.<n>Our method applies properly parameterized functionality-preserving transformations and introduces a dual learning procedure in which both models jointly learn their corresponding solutions.<n> Empirically, we show that our approach near-zero loss barriers on WikiText for language models with medium-sized parameters, representing, to our knowledge, the first demonstration of near-barrier-free linear connectivity at this scale.
  arXiv  Detail & Related papers  (2026-06-22T17:08:28Z)
• LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling [36.7482623230111]
  We present LoopMoE, a looped MoE language model that integrates sparse routing with iterative weight-shared computation.<n>Designs enable the first strictly controlled, head-to-head evaluation of a looped MoE against a Vanilla MoE.
  arXiv  Detail & Related papers  (2026-06-03T04:38:12Z)
• FLARE: Diffusion for Hybrid Language Model [72.60770374799634]
  FLARE is a systematic conversion framework for hybrid-attention large language models.<n>It combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising.<n>Our results suggest practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives.
  arXiv  Detail & Related papers  (2026-06-01T06:58:15Z)
• Generalization and Scaling Laws for Mixture-of-Experts Transformers [0.0]
  We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers.<n>We derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead.<n>We prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts.
  arXiv  Detail & Related papers  (2026-04-10T09:59:48Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations [55.047454145941366]
  Streaming Merging is an innovative model updating paradigm that conceptualizes merging as an iterative optimization process.<n> ARM is a strategy designed to approximate gradient descent dynamics.<n> ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model.
  arXiv  Detail & Related papers  (2026-02-03T08:15:57Z)
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving [9.719456684859606]
  We introduce WAM-Flow, a vision--action (VLA) model that casts ego-trajectory planning as discrete flow matching over a structured token space.<n>WAM-Flow performs fully parallel, bidirectional denoising, enabling coarse-to-fine refinement with a tunable compute-accuracy trade-off.<n>These results establish discrete flow matching as a new promising paradigm for end-to-end autonomous driving.
  arXiv  Detail & Related papers  (2025-12-05T19:36:46Z)
• DPWMixer: Dual-Path Wavelet Mixer for Long-Term Time Series Forecasting [6.01829429039985]
  Long-term time series forecasting is a critical task in computational intelligence.<n>This paper proposes DPWMixer, a computationally efficient Dual-Path architecture.<n>Experiments on eight public benchmarks demonstrate that our method achieves a consistent improvement over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-30T03:12:50Z)
• SCHEME: Scalable Channel Mixer for Vision Transformers [52.605868919281086]
  Vision Transformers have achieved impressive performance in many computation tasks.<n>We show that the dense connections can be replaced with a sparse block diagonal structure that supports larger expansion ratios.<n>We also propose the use of a lightweight, parameter-free, channel covariance attention mechanism as a parallel branch during training.
  arXiv  Detail & Related papers  (2023-12-01T08:22:34Z)
• AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation [80.33846577924363]
  We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video framegithub. It is based on two essential designs. First, we build bidirectional volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations. Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately.
  arXiv  Detail & Related papers  (2023-04-19T16:18:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.