Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

• URL: http://arxiv.org/abs/2608.14522v1
• Date: Fri, 14 Aug 2026 17:43:16 GMT
• Title: Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers
• Abstract summary: We study the three main stages of the moral AI elicitation pipeline.<n>First, morally relevant features shift across contexts.<n> preferences differ by political ideology for roughly one-third of features.<n>The wording of the elicitation question can narrow or widen ideological gaps.
• Score: 8.96645217329267
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation. In this approach, researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. Before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative ones. We examine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation. Across two phases (N = 809) in three deployment contexts (i.e., AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased), we examine the three main stages of the moral AI elicitation pipeline. First, morally relevant features shift across contexts. This suggests that feature schemas should not be assumed to transfer across deployment domains. Second, preferences differ by political ideology for roughly one-third of features, with some differences reversing direction. The ideological composition of the voter pool can therefore affect the resulting aggregated preference profile. Third, the wording of the elicitation question can narrow or widen ideological gaps by up to a full scale point. The framing conditions also change how moral foundations are associated with participants' judgments. Taken together, these findings suggest that voting-based alignment cannot deliver fair or transparent AI by aggregation alone; at minimum, each stage of the moral AI elicitation pipeline should be audited and disclosed.

Related papers

• A phenomenon of AI-conformity: how algorithms change human moral decision-making [0.0]
  The present study examined whether AI judgements affect moral decision-making in humans.<n>The results suggest that even moral judgements, despite their sensitivity and personal significance, may be susceptible to algorithmic conformity.
  arXiv  Detail & Related papers  (2026-04-13T09:09:27Z)
• The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis [33.50773360893016]
  This paper investigates how leading AI systems prioritize moral outcomes.<n>It shows that Care and Virtue values outcomes were rated most moral, while libertarian choices were consistently penalized.<n>It also highlights the need for explainability and cultural awareness as critical design principles to guide AI toward a transparent, aligned, and future symbiotic.
  arXiv  Detail & Related papers  (2025-09-12T14:37:57Z)
• A Roadmap to Pluralistic Alignment [49.29107308098236]
  We propose a roadmap to pluralistic alignment, specifically using language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems. We argue that current alignment techniques may be fundamentally limited for pluralistic AI.
  arXiv  Detail & Related papers  (2024-02-07T18:21:17Z)
• Case Repositories: Towards Case-Based Reasoning for AI Alignment [9.097877374792576]
  Case studies commonly form the pedagogical backbone in law, ethics, and many other domains. We propose a complementary approach to constitutional AI alignment that focuses on the construction of policies through judgments on a set of cases.
  arXiv  Detail & Related papers  (2023-11-18T02:02:40Z)
• Beyond Bias and Compliance: Towards Individual Agency and Plurality of Ethics in AI [0.0]
  We argue that the way data is labeled plays an essential role in the way AI behaves. We propose an alternative path that allows for the plurality of values and the freedom of individual expression.
  arXiv  Detail & Related papers  (2023-02-23T16:33:40Z)
• When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment [96.77970239683475]
  AI systems need to be able to understand, interpret and predict human moral judgments and decisions. A central challenge for AI safety is capturing the flexibility of the human moral mind. We present a novel challenge set consisting of rule-breaking question answering.
  arXiv  Detail & Related papers  (2022-10-04T09:04:27Z)
• Fairness in Agreement With European Values: An Interdisciplinary Perspective on AI Regulation [61.77881142275982]
  This interdisciplinary position paper considers various concerns surrounding fairness and discrimination in AI, and discusses how AI regulations address them. We first look at AI and fairness through the lenses of law, (AI) industry, sociotechnology, and (moral) philosophy, and present various perspectives. We identify and propose the roles AI Regulation should take to make the endeavor of the AI Act a success in terms of AI fairness concerns.
  arXiv  Detail & Related papers  (2022-06-08T12:32:08Z)
• How Different Groups Prioritize Ethical Values for Responsible AI [75.40051547428592]
  Private companies, public sector organizations, and academic groups have outlined ethical values they consider important for responsible AI technologies. While their recommendations converge on a set of central values, little is known about the values a more representative public would find important for the AI technologies they interact with and might be affected by. We conducted a survey examining how individuals perceive and prioritize responsible AI values across three groups.
  arXiv  Detail & Related papers  (2022-05-16T14:39:37Z)
• Metaethical Perspectives on 'Benchmarking' AI Ethics [81.65697003067841]
  Benchmarks are seen as the cornerstone for measuring technical progress in Artificial Intelligence (AI) research. An increasingly prominent research area in AI is ethics, which currently has no set of benchmarks nor commonly accepted way for measuring the 'ethicality' of an AI system. We argue that it makes more sense to talk about 'values' rather than 'ethics' when considering the possible actions of present and future AI systems.
  arXiv  Detail & Related papers  (2022-04-11T14:36:39Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• Artificial Intelligence, Values and Alignment [2.28438857884398]
  normative and technical aspects of the AI alignment problem are interrelated. It is important to be clear about the goal of alignment. The central challenge for theorists is not to identify 'true' moral principles for AI.
  arXiv  Detail & Related papers  (2020-01-13T10:32:16Z)
• Learning from Learning Machines: Optimisation, Rules, and Social Norms [91.3755431537592]
  It appears that the area of AI that is most analogous to the behaviour of economic entities is that of morally good decision-making. Recent successes of deep learning for AI suggest that more implicit specifications work better than explicit ones for solving such problems.
  arXiv  Detail & Related papers  (2019-12-29T17:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.