Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

• URL: http://arxiv.org/abs/2607.04371v2
• Date: Tue, 07 Jul 2026 14:13:42 GMT
• Title: Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
• Abstract summary: Nemotron-Labs-3-Puzzle-75B-A9B is a compressed variant of Nemotron-3-Super optimized for interactive deployment.<n>In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super.
• Score: 58.395498136166395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability. Our model is publicly available on Hugging Face.

Related papers

• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models [129.64558694024126]
  Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling.<n>We characterize how optimization hyper parameters, compute allocation, and architecture scale for MoE dLLMs.<n>We train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens.
  arXiv  Detail & Related papers  (2026-08-04T10:53:02Z)
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs [80.72350166388601]
  Nemotron Elastic is a framework for building reasoning-oriented LLMs.<n>It embeds nested submodels within a single parent model.<n>Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment.
  arXiv  Detail & Related papers  (2025-11-20T18:59:21Z)
• OverFill: Two-Stage Models for Efficient Language Model Decoding [68.68408155020568]
  Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs.<n>We propose OverFill, which decouples prefill and decode stages to optimize accuracy-efficiency tradeoffs.<n>Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average.
  arXiv  Detail & Related papers  (2025-08-11T20:07:34Z)
• Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs [111.69640966866059]
  Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models.<n>In this paper, we aim to uncover a recipe to harness such scale on Ascend NPUs.<n>The key goals are better usage of the computing resources under the dynamic sparse model structures and materializing the expected performance gain on the actual hardware.
  arXiv  Detail & Related papers  (2025-05-07T15:46:36Z)
• Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models [192.5620883942846]
  Nemotron-H is a family of 8B and 56B/47B hybrid Mamba-Transformer models.<n>We replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers.<n>Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized open-sourced Transformer models.
  arXiv  Detail & Related papers  (2025-04-04T17:41:58Z)
• Puzzle: Distillation-Based NAS for Inference-Optimized LLMs [17.72841008597783]
  Large language models (LLMs) offer remarkable capabilities, yet their high inference costs restrict wider adoption.<n>We present Puzzle, a hardware-aware framework that accelerates the inference of LLMs while preserving their capabilities.<n>We showcase our framework's impact via Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B) and Llama-3.3-Nemotron-49B, two publicly available models.
  arXiv  Detail & Related papers  (2024-11-28T13:45:42Z)
• 1M parameters are enough? A lightweight CNN-based model for medical image segmentation [0.4129225533930966]
  We look for a lightweight U-Net-based model which can remain the same or achieve better performance, namely U-Lite. We design U-Lite based on the principle of Depthwise Separable Convolution so that the model can both leverage the strength of CNNs and reduce a remarkable number of computing parameters. Overall, U-Lite contains only 878K parameters, 35 times less than the traditional U-Net, and much more times less than other modern Transformer-based models.
  arXiv  Detail & Related papers  (2023-06-28T11:17:37Z)
• Efficiently Scaling Transformer Inference [8.196193683641582]
  We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings. We develop a simple analytical model for inference efficiency to select the best multi-dimensional partitioning techniques optimized for TPU v4 slices. We achieve a low-batch-size latency of 29ms per token during generation (using int8 weight quantization) and a 76% MFU during large-batch-size processing of input tokens.
  arXiv  Detail & Related papers  (2022-11-09T18:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.