From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

• URL: http://arxiv.org/abs/2503.15463v2
• Date: Fri, 21 Mar 2025 10:33:21 GMT
• Title: From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
• Authors: Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, Rui Yan,
• Abstract summary: Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches.<n>This paper introduces a comprehensive framework for scalable personalized alignment of LLMs.
• Score: 41.96246165999026
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches that assume uniform human preferences, fundamentally overlooking the diversity in user values and needs. This paper introduces a comprehensive framework for scalable personalized alignment of LLMs. We establish a systematic preference space characterizing psychological and behavioral dimensions, alongside diverse persona representations for robust preference inference in real-world scenarios. Building upon this foundation, we introduce \textsc{AlignX}, a large-scale dataset of over 1.3 million personalized preference examples, and develop two complementary alignment approaches: \textit{in-context alignment} directly conditioning on persona representations and \textit{preference-bridged alignment} modeling intermediate preference distributions. Extensive experiments demonstrate substantial improvements over existing methods, with an average 17.06\% accuracy gain across four benchmarks while exhibiting a strong adaptation capability to novel preferences, robustness to limited user data, and precise preference controllability. These results validate our framework's effectiveness, advancing toward truly user-adaptive AI systems.

Related papers

• HyPerAlign: Hypotheses-driven Personalized Alignment [24.67727411391369]
  We propose a hypotheses-driven personalization approach (HyPerAlign) for large language models (LLMs) For deliberative alignment, the helpfulness of LLM models is improved by up to $70%$ on average. For authorship attribution, results indicate consistently high win-rates (commonly $>90%$) against state-of-the-art preference fine-tuning approaches.
  arXiv  Detail & Related papers  (2025-04-29T18:01:46Z)
• LoRe: Personalizing LLMs via Low-Rank Reward Modeling [47.12507639759984]
  We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions. We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks.
  arXiv  Detail & Related papers  (2025-04-20T01:16:24Z)
• A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications [28.181295575180293]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their transition to real-world applications reveals a critical limitation. This paper presents the first comprehensive survey of personalized alignment. We propose a unified framework comprising preference memory management, personalized generation, and feedback-based alignment.
  arXiv  Detail & Related papers  (2025-03-21T10:09:16Z)
• When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning [23.557084253364174]
  Reinforcement Learning from Human Feedback (RLHF) typically assumes homogeneous preferences across users, overlooking diverse human values and minority viewpoints.<n>We present a multi-faceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence.<n>These findings highlight the critical need for holistic evaluation approaches to advance the development of more effective and inclusive preference learning systems.
  arXiv  Detail & Related papers  (2025-02-26T14:14:58Z)
• Personalized Preference Fine-tuning of Diffusion Models [75.22218338096316]
  We introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences.<n>With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way.<n>Our approach achieves an average win rate of 76% over Stable Cascade, generating images that more accurately reflect specific user preferences.
  arXiv  Detail & Related papers  (2025-01-11T22:38:41Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Beyond the Binary: Capturing Diverse Preferences With Reward Regularization [15.518838657050173]
  We argue that this reliance on binary choices does not capture the broader, aggregate preferences of the target user in real-world tasks.<n>We introduce a simple yet effective method that augments existing binary preference datasets with synthetic preference judgments to estimate potential user disagreement.
  arXiv  Detail & Related papers  (2024-12-05T02:35:46Z)
• Aligning LLMs with Individual Preferences via Interaction [51.72200436159636]
  We train large language models (LLMs) that can ''interact to align'' We develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures. For evaluation, we establish the ALOE benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations.
  arXiv  Detail & Related papers  (2024-10-04T17:48:29Z)
• BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization [26.526171463511332]
  This paper examines the impact of personalized preference optimization on Large Language Models (LLMs) BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment.
  arXiv  Detail & Related papers  (2024-06-30T13:30:04Z)
• Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier [0.5120567378386615]
  We propose a unified approach to aligning large language models (LLMs) Based on a simple decomposition of preference and auxiliary objectives, we allow for tuning LLMs to optimize user and designer preferences.
  arXiv  Detail & Related papers  (2024-05-28T08:35:48Z)
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback [70.32795295142648]
  Linear alignment is a novel algorithm that aligns language models with human preferences in one single inference step. Experiments on both general and personalized preference datasets demonstrate that linear alignment significantly enhances the performance and efficiency of LLM alignment.
  arXiv  Detail & Related papers  (2024-01-21T10:46:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.