Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

• URL: http://arxiv.org/abs/2606.15007v1
• Date: Fri, 12 Jun 2026 22:56:12 GMT
• Title: Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Abstract summary: Nemotron 3 Ultra is a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model.<n>We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD)
• Score: 269.10652784788374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.

Related papers

• Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget [72.7352550413971]
  We present Tevatron 3.0, which integrates a Megatron-Core training backend into Tevatron.<n>We show that Megatron matches FSDP reranker quality and training efficiency under comparable data-parallel settings.<n>We find that the MoE reranker matches dense 8B quality while activating less than half as many parameters and achieving substantially higher inference throughput.
  arXiv  Detail & Related papers  (2026-08-02T00:55:47Z)
• Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence [215.36956014523926]
  Nemotron 3 Nano Omni is the latest model in the Nemotron multimodal series.<n>It is the first to support audio inputs alongside text, images, and video.
  arXiv  Detail & Related papers  (2026-04-27T19:49:21Z)
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning [262.0414794728708]
  We describe the pre-training, post-training, and quantization of Nemotron 3 Super.<n>Nemotron 3 Super is a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model.
  arXiv  Detail & Related papers  (2026-04-14T07:02:32Z)
• NVIDIA Nemotron 3: Efficient and Open Intelligence [227.47413816066845]
  Nemotron 3 family of models deliver strong agentic, reasoning, and conversational capabilities.<n>Nemotron 3 models are post-trained using multi-environment reinforcement learning enabling reasoning, multi-step tool use, and support granular reasoning budget control.<n>Nemotron 3 family uses a Mixture-of-Experts hybrid Mamba-Transformer architecture to provide best-in-class throughput and context lengths of up to 1M tokens.
  arXiv  Detail & Related papers  (2025-12-24T00:24:05Z)
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning [223.88531320605486]
  Nemotron 3 Nano 30B-A3B is a Mixture-of-Experts hybrid Mamba-Transformer language model.<n>Nemotron 3 Nano was pretrained on 25 trillion text tokens, including more than 3 trillion new unique tokens over Nemotron 2.
  arXiv  Detail & Related papers  (2025-12-23T23:54:32Z)
• Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data [55.65426108082807]
  We build Uni-MoE-2.0- Omni from scratch through three core contributions.<n>It is capable of omnimodal understanding, as well as generating images, text, and speech.
  arXiv  Detail & Related papers  (2025-11-16T14:10:55Z)
• Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search [42.46046429414803]
  Jet-Nemotron is a new family of hybrid-architecture language models.<n>It matches or exceeds the accuracy of leading full-attention models.
  arXiv  Detail & Related papers  (2025-08-21T17:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.