Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget

• URL: http://arxiv.org/abs/2608.00916v1
• Date: Sun, 02 Aug 2026 00:55:47 GMT
• Title: Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
• Abstract summary: We present Tevatron 3.0, which integrates a Megatron-Core training backend into Tevatron.<n>We show that Megatron matches FSDP reranker quality and training efficiency under comparable data-parallel settings.<n>We find that the MoE reranker matches dense 8B quality while activating less than half as many parameters and achieving substantially higher inference throughput.
• Score: 72.7352550413971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure available to most academic groups. Existing Tevatron reranker training relies on the Hugging Face Trainer with DeepSpeed or PyTorch FSDP1, but these backends lack efficient support for large-scale MoE training. We present Tevatron 3.0, which integrates a Megatron-Core training backend into Tevatron while preserving its data pipeline, evaluation workflow, and Hugging Face-compatible checkpoints. We benchmark existing distributed training configurations against the new backend, showing that Megatron matches FSDP reranker quality and training efficiency under comparable data-parallel settings, is up to 22% faster in the recommended single-node configuration, and supports both LoRA and full-parameter fine-tuning. Crucially, expert parallelism enables training a 30B-parameter Qwen3-30B-A3B MoE reranker, which is infeasible with PyTorch FSDP1. Using this framework, we conduct a controlled comparison of MoE versus dense models, LoRA versus full-parameter tuning, and distillation versus contrastive training on BEIR-15 with three first-stage retrievers, and report serving throughput for Hugging Face and vLLM. We find that the MoE reranker matches dense 8B quality while activating less than half as many parameters and achieving substantially higher inference throughput. We will release the framework and trained checkpoints.

Related papers

• Mach-Mind-4-Flash Technical Report [0.0]
  Mach-Mind-4-Flash is a Mixture-of-Experts (MoE) agentic model with 3B activated parameters.<n>Post-training optimization alone without scaling pre-training compute, the model achieves performance on par with or surpassing that of 100B- parameter-class models.
  arXiv  Detail & Related papers  (2026-07-10T12:57:08Z)
• Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism [11.019309466529643]
  We propose PAT, an adaptive TP method that reconfigures TP during the generation stage of each RLHF iteration.<n>Online reconfiguration mechanism updates only the states and layouts affected by TP changes.<n> PAT reduces generation latency by up to 34.6% and end-to-end RLHF training latency by up to 27.2% compared to the original VeRL setup.
  arXiv  Detail & Related papers  (2026-05-03T05:53:32Z)
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning [262.0414794728708]
  We describe the pre-training, post-training, and quantization of Nemotron 3 Super.<n>Nemotron 3 Super is a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model.
  arXiv  Detail & Related papers  (2026-04-14T07:02:32Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Continual Pre-training of MoEs: How robust is your router? [31.784662011106196]
  Mixture of Experts (MoE) transformers are promising architectures for foundation models.<n>MoEs benefit from improved sample efficiency at training time and achieve much stronger performance.<n>We show that MoEs can match the performance of a fully re-trained MoE at a fraction of the cost.
  arXiv  Detail & Related papers  (2025-03-06T22:55:01Z)
• Mixed Sparsity Training: Achieving 4$\times$ FLOP Reduction for Transformer Pretraining [32.925150708409205]
  Mixed Sparsity Training (MST) is an efficient pretraining method that can reduce about $75%$ of Floating Point Operations (FLOPs) while maintaining performance. Our experiment on GPT-2 showcases a FLOP reduction of $4times$ without compromising performance.
  arXiv  Detail & Related papers  (2024-08-21T16:13:16Z)
• Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers [107.3726071306935]
  We propose a new plug-and-play training framework, SMoE-Dropout, to enable scaling transformers to better accuracy in their full capacity without collapse. SMoE-Dropout consists of a randomly and fixed router network to activate experts and gradually increases the activated expert number as training progresses over time. Our experiments demonstrate the superior performance and substantial computation savings of SMoE-Dropout, compared to dense training baselines with equivalent parameter counts.
  arXiv  Detail & Related papers  (2023-03-02T22:12:51Z)
• Residual Mixture of Experts [75.5489156421442]
  Residual Mixture of Experts (RMoE) is an efficient training pipeline for MoE vision transformers on downstream tasks. RMoE achieves comparable results with the upper-bound MoE training, while only introducing minor additional training cost.
  arXiv  Detail & Related papers  (2022-04-20T17:29:48Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.