Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

• URL: http://arxiv.org/abs/2608.06400v1
• Date: Fri, 31 Jul 2026 17:43:30 GMT
• Title: Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
• Abstract summary: Mixture-of-Experts (MoE) reward models seek to improve interpretability by routing prompts to specialized experts.<n>We propose $textbfCo$ntribution-$textbfCo$ntrast ($textbfCo$) response-level interpretation.<n>CoCo yields more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives.
• Score: 22.088665451247753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging. Recent sparse Mixture-of-Experts (MoE) reward models seek to improve interpretability by routing prompts to specialized experts and characterizing experts through examples with high routing weights. However, routing weights only reveal which prompts an expert $\textit{receives}$, not how it $\textit{judges}$ responses, providing only a partial account of expert behavior. We therefore propose $\textbf{Co}$ntribution-$\textbf{Co}$ntrast ($\textbf{CoCo}$) response-level interpretation, which faithfully characterizes experts' roles using chosen-rejected response pairs with the largest contribution contrasts, jointly capturing routing and preference behavior. Across automatic and human evaluations, CoCo yields more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives while maintaining competitive reward modeling accuracy. To the best of our knowledge, this is the first systematic study of interpretation methods for MoE reward models.

Related papers

• Coverage-Maximizing Multinomial Subset Routing under Operational Constraints [6.499454916436427]
  We introduce Multinomial Subset Routing (MSR), a new online routing framework over $K$ experts.<n>At each round, the learner samples $M$ experts i.i.d. from the multinomial policy, and the resulting set of distinct sampled experts forms the routed subset.<n>The reward depends only on the best-performing expert(s) in the routed subset.
  arXiv  Detail & Related papers  (2026-08-17T10:30:20Z)
• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts [32.864092506780004]
  Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.<n>We introduce ProbMoE, a probabilistic routing framework that models expert selection as a distribution over cardinality-constrained expert subsets.
  arXiv  Detail & Related papers  (2026-06-01T00:16:36Z)
• SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval [61.73547199231695]
  SubSearch is a specialized framework that shifts from outcome-only supervision to intermediate reward signals.<n> Experiments show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces.<n>SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering.
  arXiv  Detail & Related papers  (2026-04-08T13:09:47Z)
• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• Upcycled and Merged MoE Reward Model for Mitigating Reward Hacking [0.0]
  We propose an upcycle and merge MoE reward modeling approach to improve robustness and efficiency of RLHF reward models.<n>Our work highlights the potential of upcycle and merge MoE structures for improving both robustness and efficiency of RLHF reward models.
  arXiv  Detail & Related papers  (2025-11-30T04:36:37Z)
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains [9.917318870162365]
  Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding.<n>Rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post-training remains underexplored.<n>We introduce RaR, an on-policy reinforcement learning method that extends RLVR beyond verifiable domains by using rubric-based feedback.
  arXiv  Detail & Related papers  (2025-07-23T17:57:55Z)
• Unified Sparse Mixture of Experts [14.774596844618396]
  Sparse Mixture of Experts (SMoEs) models scale the capacity of models while maintaining constant computational overhead.<n>This paper proposes a Unified Sparse Mixture of Experts (USMoE) framework that addresses these limitations.
  arXiv  Detail & Related papers  (2025-03-29T07:15:12Z)
• RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution [50.171320156632866]
  Reinforcement learning from human feedback offers a promising approach to aligning large language models with human preferences.<n>Current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence.<n>We propose a more fine-grained, token-level guidance approach for RL training.
  arXiv  Detail & Related papers  (2024-11-13T02:45:21Z)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization [51.98792406392873]
  Mixture of Experts (MoE) provides a powerful way to decompose dense layers into smaller, modular computations. A major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. We propose the Multilinear Mixture of Experts ($mu$MoE) layer to address this, focusing on vision models.
  arXiv  Detail & Related papers  (2024-02-19T21:20:22Z)
• On the Representation Collapse of Sparse Mixture of Experts [102.83396489230375]
  Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations. However, learning such a routing mechanism encourages token clustering around expert centroids, implying a trend toward representation collapse.
  arXiv  Detail & Related papers  (2022-04-20T01:40:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.