Co-design of LLM-based preference agents: participation may drive overtrust

• URL: http://arxiv.org/abs/2607.21757v1
• Date: Thu, 23 Jul 2026 19:19:12 GMT
• Title: Co-design of LLM-based preference agents: participation may drive overtrust
• Abstract summary: I argue that participation and process transparency can act as an "overtrust engine"<n>I develop this as a core mechanism in participatory preference agent design, treating individual alignment not as a fixed state but as an enacted process.
• Score: 0.7252027234425333
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly used to simulate human preferences in research and practical applications, raising concerns about validation, misrepresentation, and exclusion. Co-designing agents with the people they represent is a promising way to address these concerns, but participation may also mask the problems it appears to solve. This paper explores that tension through a primarily qualitative study in which 12 participants co-designed personal preference agents in the domain of household energy, via a background survey, co-design interview, and validation survey. Participants engaged readily and mostly came to see their agents as representing them well. Independent validation, however, revealed mixed human-agent alignment, with agent responses markedly more homogeneous, decisive, and abstract than the human sample. I argue that participation and process transparency can act as an "overtrust engine" that promotes trust while concealing systematic misalignment with potential structural consequences at scale. I develop this as a core mechanism in participatory preference agent design, treating individual alignment not as a fixed state but as an enacted process.

Related papers

• Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play [53.56274149236814]
  Multi-Agent Fictitious Play (MAFP) is a novel MAS paradigm that represents stakeholder stances as agents.<n>MAFP iteratively updates each agent's decision by best responding to the empirical mixture of other agents' past decisions.<n>We evaluate MAFP on challenging decision-making tasks that test the capability of deciding strategies for competitive scenarios prior to acting.
  arXiv  Detail & Related papers  (2026-06-17T17:31:06Z)
• Position: AI Agents Are Not (Yet) a Panacea for Social Simulation [62.891898598784415]
  Recent advances in large language models (LLMs) have spurred growing interest in using LLM-integrated agents for social simulation.<n>This position paper argues that LLM-based agents are not (yet) a panacea for social simulation.
  arXiv  Detail & Related papers  (2026-02-19T06:35:07Z)
• Can an Individual Manipulate the Collective Decisions of Multi-Agents? [53.01767232004823]
  M-Spoiler is a framework that simulates agent interactions within a multi-agent system to generate adversarial samples.<n>M-Spoiler introduces a stubborn agent that actively aids in optimizing adversarial samples.<n>Our findings confirm the risks posed by the knowledge of an individual agent in multi-agent systems.
  arXiv  Detail & Related papers  (2025-09-20T01:54:20Z)
• DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI [29.777890680647186]
  DoubleAgents is an agentic planning tool that embeds transparency and control through user intervention.<n>A built-in respondent simulation generates realistic scenarios, allowing users to rehearse, refine policies, and calibrate their reliance.
  arXiv  Detail & Related papers  (2025-09-16T03:43:13Z)
• Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble [46.82793004650415]
  Large language models (LLMs) have demonstrated promise in emulating human-like responses across a range of tasks.<n>We propose a novel alignment framework that treats LLMs as agent proxies for human survey respondents.<n>We introduce P2P, a system that steers LLM agents toward representative behavioral patterns using structured prompt engineering, entropy-based sampling, and regression-based selection.
  arXiv  Detail & Related papers  (2025-09-14T15:08:45Z)
• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation [18.70850695450292]
  Large Language Models (LLMs) have fueled the notion that synthetic agents can serve as substitutes for real participants in human-subject research.<n>Do agents maintain internal consistency, retaining similar behaviors when examined under different experimental settings?<n>This study explores whether an agent's conversation behavior is consistent with what we would expect from their revealed internal state.
  arXiv  Detail & Related papers  (2025-09-03T21:55:29Z)
• Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking [7.326813521586858]
  Large Language Models (LLMs) have shown strong performance across fact-checking tasks.<n>This paper investigates whether generative agents can meaningfully contribute to fact-checking tasks traditionally reserved for human crowds.<n>Agent crowds outperform human crowds in truthfulness classification, exhibit higher internal consistency, and show reduced susceptibility to social and cognitive biases.
  arXiv  Detail & Related papers  (2025-04-24T18:49:55Z)
• Language Agents as Digital Representatives in Collective Decision-Making [22.656601943922066]
  "representation" is the activity of making an individual's preferences present in the process via participation by a proxy agent.<n>We investigate the possibility of training textitlanguage agents to behave in the capacity of representatives of human agents.
  arXiv  Detail & Related papers  (2025-02-13T14:35:40Z)
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance [95.03771007780976]
  We tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions.<n>First, we collect real-world human activities to generate proactive task predictions.<n>These predictions are labeled by human annotators as either accepted or rejected.<n>The labeled data is used to train a reward model that simulates human judgment.
  arXiv  Detail & Related papers  (2024-10-16T08:24:09Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.