Many Preferences, Few Policies: Towards Scalable Language Model Personalization

• URL: http://arxiv.org/abs/2604.04144v1
• Date: Sun, 05 Apr 2026 15:12:07 GMT
• Title: Many Preferences, Few Policies: Towards Scalable Language Model Personalization
• Abstract summary: holy grail of LLM personalization is a single LLM for each user, perfectly aligned with that user's preferences.<n>We develop a principled method for selecting a small portfolio of LLMs that captures representative behaviors across heterogeneous users.<n>We provide empirical results that validate these guarantees and demonstrate greater output diversity over common baselines.
• Score: 26.263947748558824
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The holy grail of LLM personalization is a single LLM for each user, perfectly aligned with that user's preferences. However, maintaining a separate LLM per user is impractical due to constraints on compute, memory, and system complexity. We address this challenge by developing a principled method for selecting a small portfolio of LLMs that captures representative behaviors across heterogeneous users. We model user preferences across multiple traits (e.g., safety, humor, brevity) through a multi-dimensional weight vector. Given reward functions across these dimensions, our algorithm PALM (Portfolio of Aligned LLMs) generates a small portfolio of LLMs such that, for any weight vector, the portfolio contains a near-optimal LLM for the corresponding scalarized objective. To the best of our knowledge, this is the first result that provides theoretical guarantees on both the size and approximation quality of LLM portfolios for personalization. It characterizes the trade-off between system cost and personalization, as well as the diversity of LLMs required to cover the landscape of user preferences. We provide empirical results that validate these guarantees and demonstrate greater output diversity over common baselines.

Related papers

• CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM [9.154957475794612]
  We propose an interaction-efficient active learning framework in which a dueling bandit algorithm iteratively selects pairs of LLMs.<n>We experimentally verify that our model can efficiently match well-aligned LLMs to users at a lower cost.
  arXiv  Detail & Related papers  (2026-05-30T18:47:03Z)
• One Model for All: Multi-Objective Controllable Language Models [65.4626816393381]
  We introduce Multi-Objective Control (MOC) to train a single language model as a preference-conditioned policy network.<n>Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network.<n>Extensive experiments demonstrate the advantages of MOC over baselines in three aspects.
  arXiv  Detail & Related papers  (2026-04-06T07:48:32Z)
• HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation [24.67727411391369]
  HyPerAlign is an interpretable and sample-efficient hypothesis-driven personalization approach for large language models.<n>We conduct experiments on two different personalization tasks, namely authorship attribution and deliberative alignment.<n>Results demonstrate the superiority of hypothesis-driven personalization compared to preference-based fine-tuning methods.
  arXiv  Detail & Related papers  (2025-04-29T18:01:46Z)
• Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalization [68.79814761867314]
  We propose Difference-aware Personalization Learning (DPL) to enhance Large Language Models (LLMs) personalization.<n>DPL strategically selects representative users for comparison and establishes a structured standard to extract task-relevant differences.<n>Experiments on real-world datasets demonstrate that DPL significantly enhances LLM personalization.
  arXiv  Detail & Related papers  (2025-03-04T09:53:26Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models [8.558834738072363]
  Large language models (LLMs) have been widely adopted due to their remarkable performance across various applications.<n>These individual LLMs show limitations in generalization and performance on complex tasks due to inherent training biases, model size constraints, and the quality or diversity of pre-training datasets.<n>We introduce SelectLLM, which efficiently directs input queries to the most suitable subset of LLMs from a large pool.
  arXiv  Detail & Related papers  (2024-08-16T06:11:21Z)
• Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs [3.450141240227484]
  We propose a lightweight method for any-precision quantization of Large Language Models (LLMs) Our solution significantly reduces the high costs of deploying multiple, different-sized LLMs. All the supported LLMs with varying bit-widths demonstrate state-of-the-art model quality and inference throughput.
  arXiv  Detail & Related papers  (2024-02-16T09:06:06Z)
• Knowledge Fusion of Large Language Models [73.28202188100646]
  This paper introduces the notion of knowledge fusion for large language models (LLMs) We externalize their collective knowledge and unique strengths, thereby elevating the capabilities of the target model beyond those of any individual source LLM. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation.
  arXiv  Detail & Related papers  (2024-01-19T05:02:46Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.