Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation

• URL: http://arxiv.org/abs/2410.03723v1
• Date: Sun, 29 Sep 2024 04:31:45 GMT
• Title: Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation
• Authors: Tiffany Zhu, Iain Weissburg, Kexun Zhang, William Yang Wang,
• Abstract summary: This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
• Score: 48.70176791365903
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As AI advances in text generation, human trust in AI generated content remains constrained by biases that go beyond concerns of accuracy. This study explores how bias shapes the perception of AI versus human generated content. Through three experiments involving text rephrasing, news article summarization, and persuasive writing, we investigated how human raters respond to labeled and unlabeled content. While the raters could not differentiate the two types of texts in the blind test, they overwhelmingly favored content labeled as "Human Generated," over those labeled "AI Generated," by a preference score of over 30%. We observed the same pattern even when the labels were deliberately swapped. This human bias against AI has broader societal and cognitive implications, as it undervalues AI performance. This study highlights the limitations of human judgment in interacting with AI and offers a foundation for improving human-AI collaboration, especially in creative fields.

Related papers

• Measuring Human Contribution in AI-Assisted Content Generation [68.03658922067487]
  This study raises the research question of measuring human contribution in AI-assisted content generation. By calculating mutual information between human input and AI-assisted output relative to self-information of AI-assisted output, we quantify the proportional information contribution of humans in content generation.
  arXiv  Detail & Related papers  (2024-08-27T05:56:04Z)
• Rolling in the deep of cognitive and AI biases [1.556153237434314]
  We argue that there is urgent need to understand AI as a sociotechnical system, inseparable from the conditions in which it is designed, developed and deployed. We address this critical issue by following a radical new methodology under which human cognitive biases become core entities in our AI fairness overview. We introduce a new mapping, which justifies the humans to AI biases and we detect relevant fairness intensities and inter-dependencies.
  arXiv  Detail & Related papers  (2024-07-30T21:34:04Z)
• Generation Z's Ability to Discriminate Between AI-generated and Human-Authored Text on Discord [0.32885740436059047]
  Discord enables AI integrations, making their primarily "Generation Z" userbase particularly exposed to AI-generated content. We surveyed Generation Z aged individuals to evaluate their proficiency in discriminating between AI-generated and human-authored text. We find that Generation Z individuals are unable to discern between AI and human-authored text.
  arXiv  Detail & Related papers  (2023-12-31T11:52:15Z)
• Improving Human-AI Collaboration With Descriptions of AI Behavior [14.904401331154062]
  People work with AI systems to improve their decision making, but often under- or over-rely on AI predictions and perform worse than they would have unassisted. To help people appropriately rely on AI aids, we propose showing them behavior descriptions.
  arXiv  Detail & Related papers  (2023-01-06T00:33:08Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• Can Machines Imitate Humans? Integrative Turing Tests for Vision and Language Demonstrate a Narrowing Gap [45.6806234490428]
  We benchmark current AIs in their abilities to imitate humans in three language tasks and three vision tasks. Experiments involved 549 human agents plus 26 AI agents for dataset creation, and 1,126 human judges plus 10 AI judges. Results reveal that current AIs are not far from being able to impersonate humans in complex language and vision challenges.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)
• On the Influence of Explainable AI on Automation Bias [0.0]
  We aim to shed light on the potential to influence automation bias by explainable AI (XAI) We conduct an online experiment with regard to hotel review classifications and discuss first results.
  arXiv  Detail & Related papers  (2022-04-19T12:54:23Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.