PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning

• URL: http://arxiv.org/abs/2605.00384v1
• Date: Fri, 01 May 2026 04:06:44 GMT
• Title: PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
• Abstract summary: We propose PrefMoE, a mixture-of-experts reward learning framework for robust preference modeling.<n>PrefMoE learns multiple specialized reward experts and uses trajectory-level soft routing to combine them adaptively.<n>A load-balancing regularizer further stabilizes training by preventing expert collapse.
• Score: 14.1606113351064
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Preference-based reinforcement learning offers a scalable alternative to manual reward engineering by learning reward structures from comparative feedback. However, large-scale preference datasets, whether collected from crowdsourced annotators or generated by synthetic teachers, often contain heterogeneous and partially conflicting supervision, including disagreement across annotators and inconsistency within annotators. Existing reward learning methods typically fit a single reward model to such data, forcing it to average incompatible signals and thereby limiting robustness. To solve this, we propose PrefMoE, a mixture-of-experts reward learning framework for robust preference modeling. PrefMoE learns multiple specialized reward experts and uses trajectory-level soft routing to combine them adaptively, enabling the model to capture diverse latent preference patterns under noisy and heterogeneous preference supervision. A load-balancing regularizer further stabilizes training by preventing expert collapse. Across locomotion benchmarks from D4RL and manipulation tasks from MetaWorld, PrefMoE improves preference prediction robustness and leads to more reliable downstream policy learning than strong single-model baselines.

Related papers

• Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling [21.9382093355334]
  We propose a sparse Mixture-of-Experts (MoE) reward model that encourages sparse routing and expert diversity.<n>MoE learns interpretable routing patterns and specialized experts across controlled and real-world experiments.<n>It also improves test-time personalization, and post-adaptation shifts in expert weights provide a qualitative lens for analyzing how the model adapts to personalized preferences.
  arXiv  Detail & Related papers  (2026-06-02T23:19:12Z)
• Soft Specialists: $α$-Rényi Ensembles for Uncertainty-Aware LLM Post-Training [4.297453240868895]
  Existing training approaches for large language models learn a single set of parameters, based on large volumes of data.<n>We propose an $$-Rényi variational framework for learning distributions over post-training parameters.<n>We identify local stability criteria, demonstrating how model misspecification can make non-degenerate posterior spread locally favourable.
  arXiv  Detail & Related papers  (2026-05-26T22:44:58Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Optimal Transport for LLM Reward Modeling from Noisy Preference [21.518348751984433]
  We propose SelectiveRM, a framework grounded in optimal transport.<n>We first devise a Joint Consistency Discrepancy to align the distribution of model predictions with preference data.<n>We then incorporate a Mass Relaxation mechanism via partial transport.<n>This enables the autonomous exclusion of samples with noisy preference that contradict semantic consistency.
  arXiv  Detail & Related papers  (2026-05-07T11:26:04Z)
• Can Recommender Systems Teach Themselves? A Recursive Self-Improving Framework with Fidelity Control [82.30868101940068]
  We propose a paradigm in which a model bootstraps its own performance without reliance on external data or teacher models.<n>Our theoretical analysis shows that RSIR acts as a data-driven implicit regularizer, smoothing the optimization landscape.<n>We show that even smaller models benefit, and weak models can generate effective training curricula for stronger ones.
  arXiv  Detail & Related papers  (2026-02-17T15:31:32Z)
• AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling [23.81351558826977]
  We propose AdaJudge, a unified framework that jointly adapts representation and aggregation.<n>AdaJudge first refines backbone representations into a discrimination-oriented space via refinement blocks.<n>It then replaces the static readout with an adaptive multi-view pooling module that dynamically routes and combines evidence.
  arXiv  Detail & Related papers  (2026-01-13T00:37:38Z)
• Upcycled and Merged MoE Reward Model for Mitigating Reward Hacking [0.0]
  We propose an upcycle and merge MoE reward modeling approach to improve robustness and efficiency of RLHF reward models.<n>Our work highlights the potential of upcycle and merge MoE structures for improving both robustness and efficiency of RLHF reward models.
  arXiv  Detail & Related papers  (2025-11-30T04:36:37Z)
• Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation [60.33386541343322]
  We propose a Multimodal Large Language Models framework that integrates Hardness-aware and Noise-regularized preference optimization for Recommendation (HaNoRec)<n>Specifically, HaNoRec dynamically adjusts optimization weights based on both the estimated hardness of each training sample and the policy model's real-time responsiveness.
  arXiv  Detail & Related papers  (2025-11-24T04:10:46Z)
• Robust Preference Alignment via Directional Neighborhood Consensus [13.313830197011983]
  We introduce Robust Preference Selection (RPS), a post-hoc, training-free method by leveraging directional neighborhood consensus.<n>RPS samples multiple responses from a local neighborhood of related preferences to create a superior candidate pool.<n>Our work presents a practical, theoretically-grounded solution for enhancing the reliability of preference-aligned models.
  arXiv  Detail & Related papers  (2025-10-23T12:39:20Z)
• FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors [50.131271229165165]
  Federated Learning (FL) has emerged as a promising framework for distributed machine learning.<n>Data heterogeneity resulting from differences across user behaviors, preferences, and device characteristics poses a significant challenge for federated learning.<n>We propose Adaptive Weight Aggregation (FedAWA), a novel method that adaptively adjusts aggregation weights based on client vectors during the learning process.
  arXiv  Detail & Related papers  (2025-03-20T04:49:40Z)
• Best Policy Learning from Trajectory Preference Feedback [11.896067099790962]
  Preference-based Reinforcement Learning (PbRL) offers a more robust alternative.<n>We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models.<n>We propose Posterior Sampling for Preference Learning ($mathsfPSPL$), a novel algorithm inspired by Top-Two Thompson Sampling.
  arXiv  Detail & Related papers  (2025-01-31T03:55:10Z)
• Reinforcement Learning from Diverse Human Preferences [68.4294547285359]
  This paper develops a method for crowd-sourcing preference labels and learning from diverse human preferences. The proposed method is tested on a variety of tasks in DMcontrol and Meta-world. It has shown consistent and significant improvements over existing preference-based RL algorithms when learning from diverse feedback.
  arXiv  Detail & Related papers  (2023-01-27T15:18:54Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.