Auditing Preferences for Brands and Cultures in LLMs

• URL: http://arxiv.org/abs/2603.18300v1
• Date: Wed, 18 Mar 2026 21:38:39 GMT
• Title: Auditing Preferences for Brands and Cultures in LLMs
• Abstract summary: This paper introduces ChoiceEval, a reproducible framework for auditing preferences for brands and cultures in large language models (LLMs)<n> ChoiceEval addresses two core technical challenges: generating realistic, persona-diverse evaluation queries and converting free-form outputs into comparable choice sets and quantitative preference metrics.<n>It is applied to Gemini, GPT, and DeepSeek across 10 topics spanning commerce and culture and more than 2,000 questions.
• Score: 9.677509409150549
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) based AI systems increasingly mediate what billions of people see, choose and buy. This creates an urgent need to quantify the systemic risks of LLM-driven market intermediation, including its implications for market fairness, competition, and the diversity of information exposure. This paper introduces ChoiceEval, a reproducible framework for auditing preferences for brands and cultures in large language models (LLMs) under realistic usage conditions. ChoiceEval addresses two core technical challenges: (i) generating realistic, persona-diverse evaluation queries and (ii) converting free-form outputs into comparable choice sets and quantitative preference metrics. For a given topic (e.g. running shoes, hotel chains, travel destinations), the framework segments users into psychographic profiles (e.g., budget-conscious, wellness-focused, convenience), and then derives diverse prompts that reflect real-world advice-seeking and decision-making behaviour. LLM responses are converted into normalised top-k choice sets. Preference and geographic bias are then quantified using comparable metrics across topics and personas. Thus, ChoiceEval provides a scalable audit pipeline for researchers, platforms, and regulators, linking model behaviour to real-world economic outcomes. Applied to Gemini, GPT, and DeepSeek across 10 topics spanning commerce and culture and more than 2,000 questions, ChoiceEval reveals consistent preferences: U.S.-developed models Gemini and GPT show marked favouritism toward American entities, while China-developed DeepSeek exhibits more balanced yet still detectable geographic preferences. These patterns persist across user personas, suggesting systematic rather than incidental effects.

Related papers

• Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences [2.276729420256043]
  We show that aggregation masks critical information about preference diversity, individual values, and contextual dependencies.<n>We conclude with a concrete research and policy agenda for developing preference-aware models.
  arXiv  Detail & Related papers  (2026-05-30T18:47:52Z)
• Trustworthy Recommendation in the Era of Large Language Models: Opportunities and Challenges [60.18620798072947]
  Large Language Models (LLMs) have been extensively integrated into recommender systems.<n>LLMs offer unprecedented opportunities to enhance trustworthiness.<n>But they simultaneously introduce new risks, such as novel forms of bias and hallucination-induced issues.
  arXiv  Detail & Related papers  (2026-05-30T05:14:53Z)
• Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices [1.8759305308855918]
  Large Language Models (LLMs) are increasingly deployed in applications such as travel assistance and purchasing support.<n>We study LLM decision-making in a travel-assistant context by presenting models with choice dilemmas and analyzing their responses.<n>We show that while meaningful WTP values can be derived for larger LLMs, they also display systematic deviations at the attribute level.
  arXiv  Detail & Related papers  (2026-02-10T14:05:42Z)
• EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences [43.42154744611566]
  Large language models (LLMs) are deployed globally, creating pluralistic systems that can accommodate the diverse preferences of users worldwide.<n>ESTEER is a benchmark to measure models' steerability towards users' value and stylistic preference profiles.
  arXiv  Detail & Related papers  (2025-10-07T18:36:15Z)
• PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations? [0.47745223151611654]
  We propose PerFairX to quantify the trade-offs between personalization and demographic equity in LLM-generated recommendations.<n>We benchmark two state-of-the-art LLMs, ChatGPT and DeepSeek, on movie (MovieLens 10M) and music (Last.fm 360K) datasets.<n>DeepSeek achieves stronger psychological fit but exhibits higher sensitivity to prompt variations, while ChatGPT delivers stable yet less personalized outputs.
  arXiv  Detail & Related papers  (2025-08-20T09:41:53Z)
• Your AI, Not Your View: The Bias of LLMs in Investment Analysis [62.388554963415906]
  In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data.<n>These conflicts are especially problematic in real-world investment services, where a model's inherent biases can misalign with institutional objectives.<n>We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in investment analysis.
  arXiv  Detail & Related papers  (2025-07-28T16:09:38Z)
• Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? [81.49470136653665]
  We evaluate the robustness and expressiveness of value representations across three widely used probing strategies.<n>We show that the demographic context has little effect on the free-text generation, and the models' values only weakly correlate with their preference for value-based actions.
  arXiv  Detail & Related papers  (2025-07-17T18:56:41Z)
• Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models [11.585115320816257]
  This study examines the biases present in Large Language Models (LLMs) recommendations of U.S. cities and towns.<n>We focus on the consistency of LLMs responses and their tendency to over-represent or under-represent specific locations.<n>Our findings point to consistent demographic biases in these recommendations, which could perpetuate a rich-get-richer'' effect that widens existing economic disparities.
  arXiv  Detail & Related papers  (2025-03-16T18:59:00Z)
• Diverging Preferences: When do Annotators Disagree and do Models Know? [92.24651142187989]
  We develop a taxonomy of disagreement sources spanning 10 categories across four high-level classes. We find that the majority of disagreements are in opposition with standard reward modeling approaches. We develop methods for identifying diverging preferences to mitigate their influence on evaluation and training.
  arXiv  Detail & Related papers  (2024-10-18T17:32:22Z)
• Bayesian Preference Elicitation with Language Models [82.58230273253939]
  We introduce OPEN, a framework that uses BOED to guide the choice of informative questions and an LM to extract features. In user studies, we find that OPEN outperforms existing LM- and BOED-based methods for preference elicitation.
  arXiv  Detail & Related papers  (2024-03-08T18:57:52Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.