Can Revealed Preferences Clarify LLM Alignment and Steering?

• URL: http://arxiv.org/abs/2605.08556v1
• Date: Fri, 08 May 2026 23:26:35 GMT
• Title: Can Revealed Preferences Clarify LLM Alignment and Steering?
• Abstract summary: We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize.<n>We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way.
• Score: 23.175180848107093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.

Related papers

• Superficial Beliefs in LLM Decision-Making [17.52079171010035]
  We ask whether large language models (LLMs) merely imitate rationales when choosing between two options.<n>We compare the attribute a model says mattered most with the attribute that best explains its choice under a behavioural model fit to prior decisions.<n>We interpret this as evidence for superficial belief'' in LLM decision-making.
  arXiv  Detail & Related papers  (2026-06-09T15:54:35Z)
• Act or Escalate? Evaluating Escalation Behavior in Automation with Language Models [0.0]
  We model escalation as a decision under uncertainty.<n>We find marked differences in the implicit thresholds models use to trade off these costs.<n>We then test interventions that target this decision process by varying cost ratios.
  arXiv  Detail & Related papers  (2026-03-31T19:29:17Z)
• Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity [45.92643973404507]
  We investigate whether aligned models are vulnerable to Preference-Undermining Attacks (PUA), a class of manipulative prompting strategies.<n>Surprisingly, more advanced models are sometimes more susceptible to manipulative prompts.
  arXiv  Detail & Related papers  (2026-01-10T15:16:23Z)
• Judging with Confidence: Calibrating Autoraters to Preference Distributions [56.17041629492863]
  We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population.<n>We present two learning methods tailored to different data conditions.<n>Our results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution.
  arXiv  Detail & Related papers  (2025-09-30T20:36:41Z)
• DecisionFlow: Advancing Large Language Model as Principled Decision Maker [49.088778182807395]
  DecisionFlow is a novel decision modeling framework that guides models to reason over structured representations of actions, attributes, and constraints.<n>Rather than predicting answers directly from prompts, DecisionFlow builds a semantically grounded decision space and infers a latent utility function.<n> Empirical results show that DecisionFlow achieves up to 30% accuracy gains over strong prompting baselines.
  arXiv  Detail & Related papers  (2025-05-27T16:23:53Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• An exact counterfactual-example-based approach to tree-ensemble models interpretability [0.0]
  High-performance models do not exhibit the necessary transparency to make their decisions fully understandable. We could derive an exact geometrical characterisation of their decision regions under the form of a collection of multidimensional intervals. An adaptation to reasoning on regression problems is also envisaged.
  arXiv  Detail & Related papers  (2021-05-31T09:32:46Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.