Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity

• URL: http://arxiv.org/abs/2505.11861v1
• Date: Sat, 17 May 2025 06:02:00 GMT
• Title: Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity
• Authors: Qi Zhou, Jie Zhang, Dongxia Wang, Qiang Liu, Tianlin Li, Jin Song Dong, Wenhai Wang, Qing Guo,
• Abstract summary: We introduce Fair-PP, a synthetic dataset of personalized preferences targeting social equity.<n>We also contribute (i) An automated framework for generating preference data, along with a more fine-grained dataset of personalized preferences; (ii) analysis of the positioning of the existing mainstream language models across five major global regions within the personalized preference space; and (iii) a sample reweighting method for personalized preference alignment.
• Score: 33.36483739554757
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human preference plays a crucial role in the refinement of large language models (LLMs). However, collecting human preference feedback is costly and most existing datasets neglect the correlation between personalization and preferences. To address this issue, we introduce Fair-PP, a synthetic dataset of personalized preferences targeting social equity, derived from real-world social survey data, which includes 28 social groups, 98 equity topics, and 5 personal preference dimensions. Leveraging GPT-4o-mini, we engage in role-playing based on seven representative persona portrayals guided by existing social survey data, yielding a total of 238,623 preference records. Through Fair-PP, we also contribute (i) An automated framework for generating preference data, along with a more fine-grained dataset of personalized preferences; (ii) analysis of the positioning of the existing mainstream LLMs across five major global regions within the personalized preference space; and (iii) a sample reweighting method for personalized preference alignment, enabling alignment with a target persona while maximizing the divergence from other personas. Empirical experiments show our method outperforms the baselines.

Related papers

• FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data [14.12452005994486]
  We focus on a practical yet challenging setting where only a small set of preference annotations can be collected per user.<n>To support research in this area, we introduce two datasets -- DnD and ELIP.<n>We propose FaST, a highly parameter-efficient approach that leverages high-level features automatically discovered from the data.
  arXiv  Detail & Related papers  (2025-08-06T17:58:26Z)
• NextQuill: Causal Preference Modeling for Enhancing LLM Personalization [82.15961484963256]
  We introduce NextQuill, a novel personalization framework grounded in causal preference modeling.<n>Building on this insight, NextQuill introduces two complementary alignment strategies.<n> Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality.
  arXiv  Detail & Related papers  (2025-06-03T02:08:55Z)
• Anyprefer: An Agentic Framework for Preference Data Synthesis [62.3856754548222]
  We propose Anyprefer, a framework designed to synthesize high-quality preference data for aligning the target model.<n> external tools are introduced to assist the judge model in accurately rewarding the target model's responses.<n>The synthesized data is compiled into a new preference dataset, Anyprefer-V1, consisting of 58K high-quality preference pairs.
  arXiv  Detail & Related papers  (2025-04-27T15:21:59Z)
• Learning a Canonical Basis of Human Preferences from Binary Ratings [28.975782992900065]
  This paper shifts the focus to understanding the preferences encoded in such datasets and identifying common human preferences.<n>We find that a small subset of 21 preference categories captures >89% of preference variation across individuals.<n>This small set of preferences is analogous to a canonical basis of human preferences, similar to established findings that characterize human variation in psychology or facial recognition studies.
  arXiv  Detail & Related papers  (2025-03-31T14:35:48Z)
• From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment [41.96246165999026]
  Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches.<n>This paper introduces a comprehensive framework for scalable personalized alignment of LLMs.
  arXiv  Detail & Related papers  (2025-03-19T17:41:46Z)
• FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users [111.56469697145519]
  We propose Few-Shot Preference Optimization, which reframes reward modeling as a meta-learning problem.<n>Under this framework, an LLM learns to quickly adapt to a user via a few labeled preferences from that user, constructing a personalized reward function for them.<n>We generate over 1M synthetic personalized preferences using publicly available LLMs.<n>We evaluate FSPO on personalized open-ended generation for up to 1,500 synthetic users across three domains: movie reviews, pedagogical adaptation based on educational background, and general question answering, along with a controlled human study.
  arXiv  Detail & Related papers  (2025-02-26T17:08:46Z)
• Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs? [1.7819574476785418]
  This study explores the potential of Large Language Models (LLMs) to generate artificial surveys.<n>By leveraging LLMs for synthetic data creation, we aim to address the limitations of traditional survey methods.<n>A novel approach incorporating "Personas" is introduced and compared to five other synthetic survey methods.
  arXiv  Detail & Related papers  (2025-01-20T15:11:03Z)
• Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback [87.37721254914476]
  We introduce a routing framework that combines inputs from humans and LMs to achieve better annotation quality.<n>We train a performance prediction model to predict a reward model's performance on an arbitrary combination of human and LM annotations.<n>We show that the selected hybrid mixture achieves better reward model performance compared to using either one exclusively.
  arXiv  Detail & Related papers  (2024-10-24T20:04:15Z)
• ComPO: Community Preferences for Language Model Personalization [122.54846260663922]
  ComPO is a method to personalize preference optimization in language models. We collect and release ComPRed, a question answering dataset with community-level preferences from Reddit.
  arXiv  Detail & Related papers  (2024-10-21T14:02:40Z)
• Personality Alignment of Large Language Models [30.710131188931317]
  Personality Alignment aims to align large language models with individual user preferences.<n>This dataset includes data from over 320,000 real subjects across multiple personality assessments.<n>We develop an activation intervention optimization method to efficiently align with individual behavioral preferences.<n>Our work paves the way for future AI systems to make decisions and reason in truly personality ways.
  arXiv  Detail & Related papers  (2024-08-21T17:09:00Z)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models with human preferences.<n>Our key idea is leveraging the human prior knowledge within the small (seed) data.<n>We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• On Diversified Preferences of Large Language Model Alignment [51.26149027399505]
  This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them.
  arXiv  Detail & Related papers  (2023-12-12T16:17:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.