Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework

• URL: http://arxiv.org/abs/2605.10043v1
• Date: Mon, 11 May 2026 06:12:24 GMT
• Title: Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
• Abstract summary: Large Language Model (LLM) personalization aims to align model behaviors with individual user preferences.<n>Existing methods often focus on isolated user histories, neglecting the essential role of inter-user differences.<n>We propose C-BPO, a framework that personalizes LLMs via preference-calibrated binary signals.
• Score: 17.343792633458254
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM) personalization aims to align model behaviors with individual user preferences. Existing methods often focus on isolated user histories, neglecting the essential role of inter-user differences. We propose C-BPO, a framework that personalizes LLMs via preference-calibrated binary signals. By treating target user data as positive feedback and other users' data as an auxiliary set of implicit negative signals, C-BPO captures distinct inter-user differences. To mitigate the preference overlap issue, where shared task knowledge is erroneously penalized, we derive an objective grounded in Positive-Unlabeled (PU) learning theory. This approach purifies negative signals by subtracting ``positive bias'', ensuring alignment with unique idiosyncrasies without compromising general helpfulness. Empirical experiments across various personalization tasks and backbone LLMs show C-BPO consistently outperforms baselines, demonstrating the efficacy of preference-calibrated binary signals in modeling inter-user differences.

Related papers

• Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences [2.0531872026357383]
  In per-user alignment approach, preference feedback is often sparse, so learning becomes unstable and vulnerable to human preference noise.<n>A single shared policy approach to user alignment avoids this cost but fails to capture heterogeneous preferences and often neglects minority preferences.<n>We introduce Preference-based REward Clustering (PREC), a novel framework that learns a compact set of policies from binary preference labels provided by diverse users.
  arXiv  Detail & Related papers  (2026-07-14T07:50:43Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• POPI: Personalizing LLMs via Optimized Natural Language Preference Inference [42.25870704040321]
  POPI is a general framework that introduces a preference inference model to distill heterogeneous user signals into concise natural language summaries.<n>These summaries act as transparent, compact, and transferable personalization representations that condition a shared generation model to produce personalized responses.<n>Extensive experiments across four personalization benchmarks demonstrate that POPI consistently improves personalization accuracy while reducing context overhead by a large margin.
  arXiv  Detail & Related papers  (2025-10-17T23:07:57Z)
• NextQuill: Causal Preference Modeling for Enhancing LLM Personalization [82.15961484963256]
  We introduce NextQuill, a novel personalization framework grounded in causal preference modeling.<n>Building on this insight, NextQuill introduces two complementary alignment strategies.<n> Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality.
  arXiv  Detail & Related papers  (2025-06-03T02:08:55Z)
• HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation [24.67727411391369]
  HyPerAlign is an interpretable and sample-efficient hypothesis-driven personalization approach for large language models.<n>We conduct experiments on two different personalization tasks, namely authorship attribution and deliberative alignment.<n>Results demonstrate the superiority of hypothesis-driven personalization compared to preference-based fine-tuning methods.
  arXiv  Detail & Related papers  (2025-04-29T18:01:46Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• ComPO: Community Preferences for Language Model Personalization [122.54846260663922]
  ComPO is a method to personalize preference optimization in language models. We collect and release ComPRed, a question answering dataset with community-level preferences from Reddit.
  arXiv  Detail & Related papers  (2024-10-21T14:02:40Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Binary Classifier Optimization for Large Language Model Alignment [4.61411484523337]
  In real-world services such as ChatGPT, aligning models based on user feedback is crucial for improving performance.<n>Most existing alignment research relies on preference-based approaches that require both positive and negative responses as a pair.<n>We propose Binary Optimization (BCO), a technique that effectively aligns LLMs using only binary feedback.
  arXiv  Detail & Related papers  (2024-04-06T15:20:59Z)
• Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts [95.09994361995389]
  Relative Preference Optimization (RPO) is designed to discern between more and less preferred responses derived from both identical and related prompts. RPO has demonstrated a superior ability to align large language models with user preferences and to improve their adaptability during the training process.
  arXiv  Detail & Related papers  (2024-02-12T22:47:57Z)
• Bootstrapping User and Item Representations for One-Class Collaborative Filtering [24.30834981766022]
  One-class collaborative filtering (OCCF) is to identify user-item pairs that are positively-related but have not been interacted yet. This paper proposes a novel OCCF framework, named as BUIR, which does not require negative sampling.
  arXiv  Detail & Related papers  (2021-05-13T14:24:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.