GPT-4 is judged more human than humans in displaced and inverted Turing tests

• URL: http://arxiv.org/abs/2407.08853v1
• Date: Thu, 11 Jul 2024 20:28:24 GMT
• Title: GPT-4 is judged more human than humans in displaced and inverted Turing tests
• Authors: Ishika Rathi, Sydney Taylor, Benjamin K. Bergen, Cameron R. Jones,
• Abstract summary: Everyday AI detection requires differentiating between people and AI in online conversations. We measured how well people and large language models can discriminate using two modified versions of the Turing test: inverted and displaced.
• Score: 0.7437224586066946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Everyday AI detection requires differentiating between people and AI in informal, online conversations. In many cases, people will not interact directly with AI systems but instead read conversations between AI systems and other people. We measured how well people and large language models can discriminate using two modified versions of the Turing test: inverted and displaced. GPT-3.5, GPT-4, and displaced human adjudicators judged whether an agent was human or AI on the basis of a Turing test transcript. We found that both AI and displaced human judges were less accurate than interactive interrogators, with below chance accuracy overall. Moreover, all three judged the best-performing GPT-4 witness to be human more often than human witnesses. This suggests that both humans and current LLMs struggle to distinguish between the two when they are not actively interrogating the person, underscoring an urgent need for more accurate tools to detect AI in conversations.

Related papers

• Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Self-Directed Turing Test for Large Language Models [56.64615470513102]
  The Turing test examines whether AIs can exhibit human-like behaviour in natural language conversations. Traditional Turing tests adopt a rigid dialogue format where each participant sends only one message each time. This paper proposes the Self-Directed Turing Test, which extends the original test with a burst dialogue format.
  arXiv  Detail & Related papers  (2024-08-19T09:57:28Z)
• Navigating AI Fallibility: Examining People's Reactions and Perceptions of AI after Encountering Personality Misrepresentations [7.256711790264119]
  Hyper-personalized AI systems profile people's characteristics to provide personalized recommendations. These systems are not immune to errors when making inferences about people's most personal traits. We present two studies to examine how people react and perceive AI after encountering personality misrepresentations.
  arXiv  Detail & Related papers  (2024-05-25T21:27:15Z)
• People cannot distinguish GPT-4 from a human in a Turing test [0.913127392774573]
  GPT-4 was judged to be a human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%) Results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected.
  arXiv  Detail & Related papers  (2024-05-09T04:14:09Z)
• Does GPT-4 pass the Turing test? [0.913127392774573]
  The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%) We argue that the Turing test continues to be relevant as an assessment of naturalistic communication and deception.
  arXiv  Detail & Related papers  (2023-10-31T06:27:52Z)
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy [70.45420918526926]
  We present LILAC, a framework for incorporating and adapting to natural language corrections online during execution. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot. We show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users.
  arXiv  Detail & Related papers  (2023-01-06T15:03:27Z)
• Can Machines Imitate Humans? Integrative Turing Tests for Vision and Language Demonstrate a Narrowing Gap [45.6806234490428]
  We benchmark current AIs in their abilities to imitate humans in three language tasks and three vision tasks. Experiments involved 549 human agents plus 26 AI agents for dataset creation, and 1,126 human judges plus 10 AI judges. Results reveal that current AIs are not far from being able to impersonate humans in complex language and vision challenges.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)
• Human Heuristics for AI-Generated Language Are Flawed [8.465228064780744]
  We study whether verbal self-presentations, one of the most personal and consequential forms of language, were generated by AI. We experimentally demonstrate that these wordings make human judgment of AI-generated language predictable and manipulable. We discuss solutions, such as AI accents, to reduce the deceptive potential of language generated by AI.
  arXiv  Detail & Related papers  (2022-06-15T03:18:56Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)
• Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs [90.20235972293801]
  Aiming to understand how human (false-temporal)-belief-a core socio-cognitive ability unify-would affect human interactions with robots, this paper proposes to adopt a graphical model to the representation of object states, robot knowledge, and human (false-)beliefs. An inference algorithm is derived to fuse individual pg from all robots across multi-views into a joint pg, which affords more effective reasoning inference capability to overcome the errors originated from a single view.
  arXiv  Detail & Related papers  (2020-04-25T23:02:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.