CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation

• URL: http://arxiv.org/abs/2607.20862v1
• Date: Thu, 23 Jul 2026 02:39:11 GMT
• Title: CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation
• Abstract summary: Constrained Shared-Private Fusion (CSPF) is a fusion method that treats heterogeneous frozen reward models as complementary evaluators.<n>CSPF decomposes each expert signal into shared and expert-private representations, encouraging cross-expert alignment.<n>Overall, CSPF suggests that fusing hidden-state representations provides a more expressive basis for preference assessment.
• Score: 2.110528880760769
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: At present, reliable evaluation of non-verifiable tasks remains challenging. Existing approaches often fail to adequately capture the diverse evaluative criteria underlying human preferences in such tasks. To this end, we propose Constrained Shared-Private Fusion (CSPF), a fusion method that treats heterogeneous frozen reward models as complementary evaluators and learns to integrate their hidden-state representations under pairwise human-preference supervision. CSPF decomposes each expert signal into shared and expert-private representations, encouraging cross-expert alignment while preserving complementary viewpoints. Across experiments on LM-Arena target-domain adaptation and PPE out-of-distribution preference evaluation, CSPF achieves the best performance on the primary metrics among the evaluated single-expert reward-model, scalar-score multi-expert, and rubric-judge baselines. Overall, CSPF suggests that fusing hidden-state representations provides a more expressive basis for preference assessment, offering a practical route toward integrated evaluative signals for non-verifiable preference tasks.

Related papers

• Rewarding Better Thinking for LLM Preference Alignment [12.554370474428621]
  Thinking Checklist Reward (TCR) is a process-oriented reward for RL-based preference alignment.<n>TCR converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses preference-implied considerations.<n>Experiments on five models from three model families show that TCR consistently improves alignment performance across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-07-22T06:57:55Z)
• CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging [95.02210956333374]
  We propose a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation.<n>On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy.
  arXiv  Detail & Related papers  (2026-05-19T10:59:19Z)
• Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation [50.696688705287755]
  We propose Mutual Information Self-Evaluation to overcome the sparse reward challenge in reinforcement learning.<n>MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals.<n>We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy.
  arXiv  Detail & Related papers  (2026-04-13T15:18:51Z)
• Tree of Preferences for Diversified Recommendation [54.183647833064136]
  We study diversified recommendation from a data-bias perspective.<n>Inspired by the outstanding performance of large language models (LLMs) in zero-shot inference leveraging world knowledge, we propose a novel approach.
  arXiv  Detail & Related papers  (2025-12-24T04:13:17Z)
• Pluralistic Off-policy Evaluation and Alignment [47.35585359400588]
  We propose POPE, the first framework for offline pluralistic preference evaluation and alignment in LLMs.<n>POPE includes a unified reward function that combines a collaborative utility component derived from human preference signals and a diversity component inspired by entropy-based coverage measures.<n> Empirical results demonstrate that POPE efficiently enhances pluralistic response generation and maintains the models' general capabilities on downstream tasks.
  arXiv  Detail & Related papers  (2025-09-15T01:57:49Z)
• DAFE: LLM-Based Evaluation Through Dynamic Arbitration for Free-Form Question-Answering [12.879551933541345]
  We propose the Dynamic Arbitration Framework for Evaluation (DAFE) to evaluate large language models.<n>DAFE employs two primary LLM-as-judges and engages a third arbitrator only in cases of disagreements.<n>We show DAFE's ability to provide consistent, scalable, and resource-efficient assessments.
  arXiv  Detail & Related papers  (2025-03-11T15:29:55Z)
• HREF: Human Response-Guided Evaluation of Instruction Following in Language Models [61.273153125847166]
  We develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF)<n>In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination.<n>We study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template.
  arXiv  Detail & Related papers  (2024-12-20T03:26:47Z)
• CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System [16.84754752395103]
  This work takes a critical stance on previous studies concerning fairness evaluation in Large Language Model (LLM)-based recommender systems.<n>We introduce CFaiRLLM, an enhanced evaluation framework that not only incorporates true preference alignment but also rigorously examines intersectional fairness.<n>To validate the efficacy of CFaiRLLM, we conducted extensive experiments using MovieLens and LastFM.
  arXiv  Detail & Related papers  (2024-03-08T20:44:59Z)
• HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition [92.17397504834825]
  HD-Eval is a framework that iteratively aligns large language models evaluators with human preference. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators.
  arXiv  Detail & Related papers  (2024-02-24T08:01:32Z)
• FedSPLIT: One-Shot Federated Recommendation System Based on Non-negative Joint Matrix Factorization and Knowledge Distillation [7.621960305708476]
  We present the first unsupervised one-shot federated CF implementation, named FedSPLIT, based on NMF joint factorization. FedSPLIT can obtain similar results than the state of the art (and even outperform it in certain situations) with a substantial decrease in the number of communications.
  arXiv  Detail & Related papers  (2022-05-04T23:42:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.