P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling

• URL: http://arxiv.org/abs/2602.12116v1
• Date: Thu, 12 Feb 2026 16:07:22 GMT
• Title: P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling
• Authors: Pinyi Zhang, Ting-En Lin, Yuchuan Wu, Jingyang Chen, Zongqi Wang, Hua Yang, Ze Xu, Fei Huang, Kai Zhang, Yongbin Li,
• Abstract summary: We propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling.<n>P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics.<n>It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism.
• Score: 66.55381105691818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.

Related papers

• Synthetic Interaction Data for Scalable Personalization in Large Language Models [67.31884245564086]
  We introduce a high-fidelity synthetic data generation framework called PersonaGym.<n>Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process.<n>We release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories.
  arXiv  Detail & Related papers  (2026-02-12T20:41:22Z)
• CURP: Codebook-based Continuous User Representation for Personalized Generation with LLMs [60.867541073274715]
  We propose a novel framework CURP, which employs a bidirectional user encoder and a discrete prototype codebook to extract multi-dimensional user traits.<n>This design enables plug-and-play personalization with a small number of trainable parameters.<n>We show that CURP achieves superior performance and generalization compared to strong baselines.
  arXiv  Detail & Related papers  (2026-01-31T14:13:06Z)
• One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment [55.86333374784959]
  We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation.<n>We propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem.<n>We show that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines.
  arXiv  Detail & Related papers  (2026-01-26T17:55:52Z)
• Lightweight Inference-Time Personalization for Frozen Knowledge Graph Embeddings [0.0]
  GatedBias is a lightweight inference-time personalization framework for knowledge graphs.<n>Profile-specific features combine with graph-derived binary gates to produce interpretable, per-entity biases.<n>We evaluate GatedBias on two benchmark datasets.
  arXiv  Detail & Related papers  (2025-12-26T22:30:37Z)
• Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learning [19.994184617064395]
  We propose a unified framework that models both individual- and group-level preferences for user satisfaction estimation.<n>Experiments on the Emotional Support Conversation dataset demonstrate consistent improvements in user satisfaction estimation.
  arXiv  Detail & Related papers  (2025-11-07T16:34:03Z)
• Conv4Rec: A 1-by-1 Convolutional AutoEncoder for User Profiling through Joint Analysis of Implicit and Explicit Feedbacks [35.7275102787435]
  We introduce a new convolutional AutoEncoder architecture for user modelling and recommendation tasks.<n>Our model is able to learn jointly from both the explicit ratings and the implicit information in the sampling pattern.<n>In experiments on several real-life datasets, we achieve state-of-the-art performance on both the implicit and explicit feedback prediction tasks.
  arXiv  Detail & Related papers  (2025-09-09T08:25:11Z)
• LoRe: Personalizing LLMs via Low-Rank Reward Modeling [47.12507639759984]
  We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions.<n>We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks.
  arXiv  Detail & Related papers  (2025-04-20T01:16:24Z)
• From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment [41.96246165999026]
  Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches.<n>This paper introduces a comprehensive framework for scalable personalized alignment of LLMs.
  arXiv  Detail & Related papers  (2025-03-19T17:41:46Z)
• Test-Time Alignment via Hypothesis Reweighting [56.71167047381817]
  Large pretrained models often struggle with underspecified tasks.<n>We propose a novel framework to address the challenge of aligning models to test-time user intent.
  arXiv  Detail & Related papers  (2024-12-11T23:02:26Z)
• Unsupervised Model Personalization while Preserving Privacy and Scalability: An Open Problem [55.21502268698577]
  This work investigates the task of unsupervised model personalization, adapted to continually evolving, unlabeled local user images. We provide a novel Dual User-Adaptation framework (DUA) to explore the problem. This framework flexibly disentangles user-adaptation into model personalization on the server and local data regularization on the user device.
  arXiv  Detail & Related papers  (2020-03-30T09:35:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.