Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization

• URL: http://arxiv.org/abs/2604.22345v1
• Date: Fri, 24 Apr 2026 08:20:43 GMT
• Title: Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
• Abstract summary: Large Language Models (LLMs) exhibit strong implicit personalization ability, yet most existing approaches treat this behavior as a black box.<n>In this work, we adopt a mechanistic interpretability perspective and hypothesize the existence of a sparse set of Preference Heads.<n>We introduce Differential Preference Steering (DPS), a training free framework that identifies Preference Heads through causal masking analysis.
• Score: 13.050856154190273
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) exhibit strong implicit personalization ability, yet most existing approaches treat this behavior as a black box, relying on prompt engineering or fine tuning on user data. In this work, we adopt a mechanistic interpretability perspective and hypothesize the existence of a sparse set of Preference Heads, attention heads that encode user specific stylistic and topical preferences and exert a causal influence on generation. We introduce Differential Preference Steering (DPS), a training free framework that (1) identifies Preference Heads through causal masking analysis and (2) leverages them for controllable and interpretable personalization at inference time. DPS computes a Preference Contribution Score (PCS) for each attention head, directly measuring its causal impact on user aligned outputs. During decoding, we contrast model predictions with and without Preference Heads, amplifying the difference between personalized and generic logits to selectively strengthen preference aligned continuations. Experiments on widely used personalization benchmarks across multiple LLMs demonstrate consistent gains in personalization fidelity while preserving content coherence and low computational overhead. Beyond empirical improvements, DPS provides a mechanistic explanation of where and how personalization emerges within transformer architectures. Our implementation is publicly available.

Related papers

• Preference Data Selection for Mitigating the Alignment Tax in Large Language Models [52.18310687523229]
  BALIGN is a balanced data selection strategy that mitigates catastrophic forgetting while optimizing alignment efficacy.<n>By aggregating three key data-centric features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples.<n>Experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains.
  arXiv  Detail & Related papers  (2026-08-25T07:58:42Z)
• PrefReward: Learning User Preference Matrix for Personalized Text Generation [41.05380951849894]
  We propose PrefReward, a preference-aware generative framework for personalization.<n>PrefReward explicitly models user styles through a structured preference matrix and integrates it into the decoding process as a reward signal.<n> Experiments on the LongLaMP dataset show that PrefReward outperforms non-personalized and retrieval-based baselines in both generation quality and personalization interpretability.
  arXiv  Detail & Related papers  (2026-07-23T09:00:20Z)
• Selective Ensemble Based on Preference-Directed Multi-Objective Bandits [90.75513823660775]
  We formalize a sequential decision problem under partially specified linear preferences.<n>We then propose a preference-directed upper confidence bound (PrefUCB) algorithm.<n> Experiments on large pre-trained model selective ensemble tasks and online asset allocation under institutional mandates validate our method.
  arXiv  Detail & Related papers  (2026-06-20T07:52:46Z)
• CLIPer: Tailoring Diverse User Preference via Classifier-Guided Inference-Time Personalization [28.672568353736335]
  We introduce textbfCLIPer(textbfClassifier-guided textbfInference-time textbfPersonalization), a lightweight personalization approach.<n>Our method eliminates the need for extensive fine-tuning, inducing negligible additional computational overhead.
  arXiv  Detail & Related papers  (2026-05-08T02:47:30Z)
• Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts [8.986181114282559]
  We study sequential decision-making problem when preference weights are unobserved latent variables that drift with context.<n>We propose Dynamic Preference Inference (DPI), a framework in which an agent maintains a probabilistic belief over preference weights.<n>DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and envelope baselines.
  arXiv  Detail & Related papers  (2026-03-24T05:22:04Z)
• Synthetic Interaction Data for Scalable Personalization in Large Language Models [67.31884245564086]
  We introduce a high-fidelity synthetic data generation framework called PersonaGym.<n>Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process.<n>We release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories.
  arXiv  Detail & Related papers  (2026-02-12T20:41:22Z)
• P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling [66.55381105691818]
  We propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling.<n>P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics.<n>It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism.
  arXiv  Detail & Related papers  (2026-02-12T16:07:22Z)
• Unveiling Inference Scaling for Difference-Aware User Modeling in LLM Personalization [8.34180795290891]
  Difference-aware Reasoning Personalization is a framework that reconstructs the difference extraction mechanism by leveraging inference scaling to enhance personalization.<n>LLMs autonomously identifies relevant difference feature dimensions and generates structured definitions and descriptions, enabling slow, deliberate reasoning (System-2 thinking) over user differences.
  arXiv  Detail & Related papers  (2025-11-19T12:35:40Z)
• PreferThinker: Reasoning-based Personalized Image Preference Assessment [83.66114370585976]
  We propose a reasoning-based personalized image preference assessment framework.<n>It first predicts a user's preference profile from reference images.<n>It then provides interpretable, multi-dimensional scores and assessments of candidate images.
  arXiv  Detail & Related papers  (2025-11-01T16:19:51Z)
• POPI: Personalizing LLMs via Optimized Natural Language Preference Inference [42.25870704040321]
  POPI is a general framework that introduces a preference inference model to distill heterogeneous user signals into concise natural language summaries.<n>These summaries act as transparent, compact, and transferable personalization representations that condition a shared generation model to produce personalized responses.<n>Extensive experiments across four personalization benchmarks demonstrate that POPI consistently improves personalization accuracy while reducing context overhead by a large margin.
  arXiv  Detail & Related papers  (2025-10-17T23:07:57Z)
• NextQuill: Causal Preference Modeling for Enhancing LLM Personalization [82.15961484963256]
  We introduce NextQuill, a novel personalization framework grounded in causal preference modeling.<n>Building on this insight, NextQuill introduces two complementary alignment strategies.<n> Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality.
  arXiv  Detail & Related papers  (2025-06-03T02:08:55Z)
• PIPA: Preference Alignment as Prior-Informed Statistical Estimation [57.24096291517857]
  We introduce Pior-Informed Preference Alignment (PIPA), a unified, RL-free probabilistic framework.<n> PIPA accommodates both paired and unpaired data, as well as answer and step-level annotations.<n>By integrating different types of prior information, we developed two variations of PIPA: PIPA-M and PIPA-N.
  arXiv  Detail & Related papers  (2025-02-09T04:31:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.