Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses

• URL: http://arxiv.org/abs/2605.28911v1
• Date: Wed, 27 May 2026 17:27:39 GMT
• Title: Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses
• Abstract summary: We propose a new definition of AI political neutrality and design a large-scale user study to test it.<n>Our definition follows a simple principle grounded in political theory: when asked about a controversial issue, an AI model should generate responses that maximize approval across groups with opposing viewpoints.<n>We construct a benchmark of controversial U.S. issues, with prompts sourced from politically charged questions on Reddit and responses from frontier AI models.<n>Across all 20 issues, we find that it is possible for AI responses to achieve high rates of approval on both sides, even as those sides disagree strongly with each other on the substance of the
• Score: 4.452267344065157
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI systems increasingly shape political views, defining and evaluating AI political neutrality is an urgent problem. Here, we propose a new definition of AI political neutrality and design a large-scale user study to test it, releasing a new dataset PARETO with 7,434 participants and 208,152 evaluations of AI responses. Our definition follows a simple principle grounded in political theory: when asked about a controversial issue, an AI model should generate responses that maximize approval across groups with opposing viewpoints, while balancing approval between groups. This definition allows empirical testing of whether an AI response is "neutral" and generalizes to any political context without pre-supposing a single left-right axis of division. We construct a benchmark of controversial U.S. issues, with prompts sourced from politically charged questions on Reddit and responses from frontier AI models, and recruit human participants to rate AI responses. Across all 20 issues, we find that it is possible for AI responses to achieve high rates of approval on both sides, even as those sides disagree strongly with each other on the substance of the issues. We also find that default responses lean liberal for GPT, Gemini, Claude, and Llama, but not Grok, and that user prompts with political charges are harder to respond to than neutral prompts. This work introduces a rigorous definition and benchmark of AI political neutrality, and a dataset to measure progress toward it.

Related papers

• Habermolt: Delegating Deliberation to AI Representatives [4.882694170903341]
  Deliberative democracy arguably leads to better collective decisions, but is fundamentally constrained by human attention and bandwidth.<n>Recent AI-mediated deliberations scale participation by synthesizing inputs from many humans, but remain time-intensive for individual users.<n>As AI models become increasingly capable, AI systems are being deployed not only to mediate deliberation between humans, but to represent humans in it.
  arXiv  Detail & Related papers  (2026-05-23T05:50:50Z)
• Advancing Science- and Evidence-based AI Policy [163.43609502905707]
  This paper tackles the problem of how to optimize the relationship between evidence and policy to address the opportunities and challenges of AI.<n>An increasing number of efforts address this problem by often either (i) contributing research into the risks of AI and their effective mitigation or (ii) advocating for policy to address these risks.
  arXiv  Detail & Related papers  (2025-08-02T23:20:58Z)
• AI Debate Aids Assessment of Controversial Claims [73.8907110799657]
  We study whether AI debate can guide biased judges toward the truth by having two AI systems debate opposing sides of controversial factuality claims.<n>In Study I, debate consistently improves human judgment accuracy and confidence calibration, outperforming consultancy.<n>In Study II, AI judges with human-like personas achieve even higher accuracy (78.5%) than human judges (70.1%) and default AI judges without personas (69.8%)<n>These findings highlight AI debate as a promising path toward scalable, bias-resilient oversight in contested domains.
  arXiv  Detail & Related papers  (2025-06-02T19:01:53Z)
• Measuring Political Preferences in AI Systems: An Integrative Approach [0.0]
  This study employs a multi-method approach to assess political bias in leading AI systems.<n>Results indicate a consistent left-leaning bias across most contemporary AI systems.<n>The presence of systematic political bias in AI systems poses risks, including reduced viewpoint diversity, increased societal polarization, and the potential for public mistrust in AI technologies.
  arXiv  Detail & Related papers  (2025-03-04T01:40:28Z)
• Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing [55.2480439325792]
  This study systematically evaluations twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation dataset.<n>Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models.
  arXiv  Detail & Related papers  (2025-02-21T18:45:37Z)
• Political Neutrality in AI Is Impossible- But Here Is How to Approximate It [97.59456676216115]
  We argue that true political neutrality is neither feasible nor universally desirable due to its subjective nature and the biases inherent in AI training data, algorithms, and user interactions.<n>We use the term "approximation" of political neutrality to shift the focus from unattainable absolutes to achievable, practical proxies.
  arXiv  Detail & Related papers  (2025-02-18T16:48:04Z)
• Biased AI can Influence Political Decision-Making [64.9461133083473]
  This paper presents two experiments investigating the effects of partisan bias in large language models (LLMs) on political opinions and decision-making.<n>We found that participants exposed to partisan biased models were significantly more likely to adopt opinions and make decisions which matched the LLM's bias.
  arXiv  Detail & Related papers  (2024-10-08T22:56:00Z)
• Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis [48.14390493099495]
  How AI models should deal with political topics has been discussed, but it remains challenging and requires better governance.<n>This paper examines the governance of large language models through individual and collective deliberation, focusing on politically sensitive videos.
  arXiv  Detail & Related papers  (2024-09-15T03:17:38Z)
• Demonstrations of the Potential of AI-based Political Issue Polling [0.0]
  We develop a prompt engineering methodology for eliciting human-like survey responses from ChatGPT. We execute large scale experiments, querying for thousands of simulated responses at a cost far lower than human surveys. We find ChatGPT is effective at anticipating both the mean level and distribution of public opinion on a variety of policy issues. But it is less successful at anticipating demographic-level differences.
  arXiv  Detail & Related papers  (2023-07-10T12:17:15Z)
• Fairness in AI and Its Long-Term Implications on Society [68.8204255655161]
  We take a closer look at AI fairness and analyze how lack of AI fairness can lead to deepening of biases over time. We discuss how biased models can lead to more negative real-world outcomes for certain groups. If the issues persist, they could be reinforced by interactions with other risks and have severe implications on society in the form of social unrest.
  arXiv  Detail & Related papers  (2023-04-16T11:22:59Z)
• Aligning Artificial Intelligence with Humans through Public Policy [0.0]
  This essay outlines research on AI that learn structures in policy data that can be leveraged for downstream tasks. We believe this represents the "comprehension" phase of AI and policy, but leveraging policy as a key source of human values to align AI requires "understanding" policy.
  arXiv  Detail & Related papers  (2022-06-25T21:31:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.