On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

• URL: http://arxiv.org/abs/2608.30320v1
• Date: Mon, 31 Aug 2026 06:35:07 GMT
• Title: On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
• Abstract summary: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
• Score: 57.03393882933515
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.

Related papers

• Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems [0.2730969268472861]
  Kolmogorov-Arnold Networks (KANs) replace the fixed activation functions and linear weights of Multi-Layer Perceptrons (MLPs)<n>This paper presents an empirical scalability study of data-parallel KAN training on multi-node, multi-GPU high-performance computing infrastructure.<n>Experiments were conducted on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel (DDP)
  arXiv  Detail & Related papers  (2026-09-07T16:45:20Z)
• p-Spin Glass Network Efficient Single-Batch Continual Learning [0.0]
  We introduce the $p$-Spin Glass Network, a novel architecture that overcomes limitations, structurally manages optimization variance and yields four noticeable capabilities.<n>It demonstrates sample efficiency, matching the performance of a Transformer baseline while utilizing $8times$ fewer training sequences.<n>Ultimately, this work removes large requirement for stable deep learning, establishing a foundation for continuous learning and edge AI.
  arXiv  Detail & Related papers  (2026-08-14T16:24:37Z)
• Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks [2.179313476241343]
  We investigate the architectural design space of Conal Neural Networks within the Parallelized Q-Network (PQN) algorithm.<n>We study the effect of multiplicative representation learning and advanced value estimation by integrating the Hadamax encoding paradigm with categorical, ensemble, and dueling value heads.<n>The results demonstrate that carefully designed encoder topology, multiplicative feature interactions, and advanced value-estimation heads can substantially improve performance within a parallelized, replay-free Q-learning framework.
  arXiv  Detail & Related papers  (2026-08-07T15:29:15Z)
• GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning [61.67411833252235]
  We introduce GradCuit, which inserts optimizable latent states at a selected Transformer layer between the hidden representations of the prompt and the generated continuation.<n>Across five instruction-tuned backbones, three reasoning benchmarks, and two answer formats, GradCuit achieves an average accuracy of 64.5%.<n>For interpretability, token-level gradient attribution reveals that latent influence concentrates on reasoning-connector tokens, while layer analysis identifies early-to-middle Transformer layers as the most effective optimization space.
  arXiv  Detail & Related papers  (2026-08-03T17:55:24Z)
• Stable FP4 Training via Transposition-Invariant Block Quantization [14.373963304887754]
  In conventional 1D block quantization, forward and backward passes assign di erent scaling factors to the same values after transposition.<n>We propose a low-precision training framework based on 2D block FP4 quantization, which enforces transposition-invariant scaling and preserves consistency between forward and backward computations.<n>Our approach achieves stable end-to-end FP4 training and closely matches BF16 performance, with less than 1.3% degradation in perplexity and downstream accuracy.
  arXiv  Detail & Related papers  (2026-07-27T18:03:34Z)
• Forge-UGC: FX optimization and register-graph engine for universal graph compiler [2.587194279527956]
  Existing frameworks such as OpenVINO and ONNX often use opaque compilation pipelines.<n> Forge-UGC addresses this with a hardware-agnostic design that separates graph capture, optimization, intermediate representation lowering, and backend scheduling.
  arXiv  Detail & Related papers  (2026-04-14T04:39:16Z)
• Gemma 4, Phi-4, and Qwen3: Accuracy-Efficiency Tradeoffs in Dense and MoE Reasoning Language Models [6.396911723204044]
  Mixture-of-experts (MoE) language models are often expected to offer better quality-efficiency tradeoffs than dense models.<n>We present a benchmark of seven recent reasoning-oriented instruction-tuned models spanning dense and MoE designs.
  arXiv  Detail & Related papers  (2026-04-08T12:50:52Z)
• Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters [169.7981969517903]
  Step 3.5 Flash bridges frontier-level agentic intelligence and computational efficiency.<n>We focus on what matters most when building agents: sharp reasoning and fast, reliable execution.
  arXiv  Detail & Related papers  (2026-02-11T07:53:51Z)
• BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity [66.94629945519125]
  We introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques.<n>Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing.<n>Next, to promote both token-level sparsity (TLS) and chunk-level sparsity ( CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly.
  arXiv  Detail & Related papers  (2025-07-11T17:28:56Z)
• A Stable Whitening Optimizer for Efficient Neural Network Training [99.7641280234926]
  Building on the Shampoo family of algorithms, we identify and alleviate three key issues, resulting in the proposed SPlus method.<n>First, we find that naive Shampoo is prone to divergence when matrix-inverses are cached for long periods.<n>Second, we adapt a shape-aware scaling to enable learning rate transfer across network width.<n>Third, we find that high learning rates result in large parameter noise, and propose a simple iterate-averaging scheme which unblocks faster learning.
  arXiv  Detail & Related papers  (2025-06-08T18:43:31Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.