Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts

• URL: http://arxiv.org/abs/2609.01100v1
• Date: Tue, 01 Sep 2026 11:43:41 GMT
• Title: Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
• Abstract summary: We show that Contrastive Routing Mechanism (CoRM) scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state.<n>Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines.
• Score: 7.00753296402747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rather than routing on absolute magnitude, concentrates the routing signal onto a low-dimensional, highly separable subspace. Building on this, we propose the Contrastive Routing Mechanism (CoRM), which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection. The resulting experts have routing boundaries that align with linguistic structure significantly more than the Top-k baseline. Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines on nine zero-shot reasoning benchmarks, at the minimal cost of 2.9% added parameters and 2.6% added FLOPs per token.

Related papers

• ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs [15.24786135265898]
  Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs)<n>We propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping.
  arXiv  Detail & Related papers  (2026-09-04T14:56:00Z)
• CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation [49.0564416580726]
  We distill the compositional judgments of a cross-attentive reranker into an embedding model.<n>We show that CORE-RERANKER-8B achieves an 82.7% total average, outperforming Jina-Reranker by 10.7 points.<n>The improvements transfer to the MCMR benchmark without sacrificing retrieval performance on COCO and Flickr30K.
  arXiv  Detail & Related papers  (2026-09-03T16:50:29Z)
• Strength-Parity Ensembling with Parameter-Isolated Experts for Multi-Task Affect Recognition [4.4644619064521995]
  We build on a shared affect-latent that marginalizes the missing labels across two affect-supervised backbones.<n>An added member lowers the ensemble error only when it is both decorrelated from the current members and a near-peer of them in individual accuracy.<n>The resulting system raises the overall validation score to 1.6949, against the organizersNeXt-with-Mixment baseline of 0.45.
  arXiv  Detail & Related papers  (2026-07-12T17:12:15Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• A Modelling and Evaluation Framework for EuroCrops-Driven Sentinel-2 Crop Segmentation [78.66324246922831]
  This work presents a pipeline for generating semantic-segmentation-ready agricultural datasets from Sentinel-2 imagery and EuroCrops parcel-level annotations.<n>The dataset contains 67,337 patches from five European countries and uses a reduced taxonomy of ten crop classes plus background.<n>A four-level U-Net with Group Normalization was trained using 10 Sentinel-2 spectral bands and a composite loss combining class-weighted cross-entropy and Dice loss.
  arXiv  Detail & Related papers  (2026-05-30T11:20:29Z)
• Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation [24.67838359287715]
  We introduce PPaint, a matched dual-protocol benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings.<n>We collect 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings.<n>Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth.
  arXiv  Detail & Related papers  (2026-05-19T12:44:01Z)
• Hierarchical Mixture-of-Experts with Two-Stage Optimization [84.70724165894501]
  We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels.<n>We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks.<n>In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B.
  arXiv  Detail & Related papers  (2026-05-08T09:21:46Z)
• MARCO: Navigating the Unseen Space of Semantic Correspondence [28.55177516749558]
  MARCO is a unified model for generalizable correspondence driven by a novel training framework.<n>Our approach transforms a handful of keypoints into dense, semantically coherent correspondences.<n>MARCO sets a new state of the art on SPair-71k, AP-10K, and PF-PASCAL.
  arXiv  Detail & Related papers  (2026-04-20T13:44:46Z)
• Milestones over Outcome: Unlocking Geometric Reasoning with Sub-Goal Verifiable Reward [67.00373428443879]
  We introduce a paradigm shift towards subgoal-level evaluation and learning.<n>We first construct GeoGoal, a benchmark synthesized via a rigorous formal verification data engine.<n>We propose the Sub-Goal Verifiable Reward (SGVR) framework, which replaces sparse signals with dense rewards based on the Skeleton Rate.
  arXiv  Detail & Related papers  (2026-01-08T16:17:56Z)
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning [49.16227597771663]
  D2Pruner is a framework that combines debiased importance with a structural pruning mechanism.<n>It reduces FLOPs by 74.2% while retaining 99.2% of its original performance.<n>It marks a significant advancement with up to 63. 53% improvement over existing methods.
  arXiv  Detail & Related papers  (2025-12-22T14:42:31Z)
• The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation [2.497854684676663]
  We present a comprehensive study contrasting a fine-tuned GPT-2 large (774M) model against the GPT-2 small (124M) model and traditional LSTM/RNN baselines on the 5-cuisine corpus from RecipeDB.<n>Our key contribution is a targeted tokenization strategy that augments the vocabulary with 23 common fraction tokens and custom structural markers.
  arXiv  Detail & Related papers  (2025-08-20T13:53:13Z)
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts [63.67734699877724]
  MoE++ is a general and heterogeneous MoE framework that integrates both Feed-Forward Network(FFN) and zero-computation experts. MoE++ achieves better performance while delivering 1.1-2.1x expert forward throughput compared to a vanilla MoE model of the same size.
  arXiv  Detail & Related papers  (2024-10-09T18:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.