Human or Not? A Gamified Approach to the Turing Test

• URL: http://arxiv.org/abs/2305.20010v1
• Date: Wed, 31 May 2023 16:32:22 GMT
• Title: Human or Not? A Gamified Approach to the Turing Test
• Authors: Daniel Jannai, Amos Meron, Barak Lenz, Yoav Levine, Yoav Shoham
• Abstract summary: We present "Human or Not?", an online game inspired by the Turing test. The game was played by over 1.5 million users over a month. Overall users guessed the identity of their partners correctly in only 68% of the games. In the subset of the games in which users faced an AI bot, users had even lower correct guess rates of 60%.
• Score: 11.454575816255039
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present "Human or Not?", an online game inspired by the Turing test, that measures the capability of AI chatbots to mimic humans in dialog, and of humans to tell bots from other humans. Over the course of a month, the game was played by over 1.5 million users who engaged in anonymous two-minute chat sessions with either another human or an AI language model which was prompted to behave like humans. The task of the players was to correctly guess whether they spoke to a person or to an AI. This largest scale Turing-style test conducted to date revealed some interesting facts. For example, overall users guessed the identity of their partners correctly in only 68% of the games. In the subset of the games in which users faced an AI bot, users had even lower correct guess rates of 60% (that is, not much higher than chance). This white paper details the development, deployment, and results of this unique experiment. While this experiment calls for many extensions and refinements, these findings already begin to shed light on the inevitable near future which will commingle humans and AI.

Related papers

• Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• The Impacts of AI Avatar Appearance and Disclosure on User Motivation [0.0]
  This study examines the influence of perceived AI features on user motivation in virtual interactions. We conducted a game-based experiment involving over 72,500 participants who solved search problems alone or with an AI companion.
  arXiv  Detail & Related papers  (2024-07-31T10:48:55Z)
• GPT-4 is judged more human than humans in displaced and inverted Turing tests [0.7437224586066946]
  Everyday AI detection requires differentiating between people and AI in online conversations. We measured how well people and large language models can discriminate using two modified versions of the Turing test: inverted and displaced.
  arXiv  Detail & Related papers  (2024-07-11T20:28:24Z)
• Prompting Fairness: Artificial Intelligence as Game Players [0.0]
  Utilitarian games to measure fairness have been studied in the social sciences for decades. Over 101 rounds of the dictator game, I conclude that AI has a strong sense of fairness that is dependant on it. There may be evidence that AI experiences inequality aversion just as humans.
  arXiv  Detail & Related papers  (2024-02-08T16:24:40Z)
• A Turing Test: Are AI Chatbots Behaviorally Similar to Humans? [19.50537882161282]
  ChatGPT-4 exhibits and personality traits that are statistically indistinguishable from a random human subjects. Their behaviors are often distinct from average and human behaviors. We estimate that they act as if they are maximizing an average of their own and partner's payoffs.
  arXiv  Detail & Related papers  (2023-11-19T16:44:09Z)
• Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games [36.96985093527702]
  We collect hundreds of crowd-sourced assessments comparing the human-likeness of navigation behavior generated by our agent and baseline AI agents. Our proposed agent passes a Turing Test, while the baseline agents do not. This work provides insights into the characteristics that people consider human-like in the context of goal-directed video game navigation.
  arXiv  Detail & Related papers  (2023-03-02T18:59:04Z)
• Can Machines Imitate Humans? Integrative Turing Tests for Vision and Language Demonstrate a Narrowing Gap [45.6806234490428]
  We benchmark current AIs in their abilities to imitate humans in three language tasks and three vision tasks. Experiments involved 549 human agents plus 26 AI agents for dataset creation, and 1,126 human judges plus 10 AI judges. Results reveal that current AIs are not far from being able to impersonate humans in complex language and vision challenges.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)
• DanZero: Mastering GuanDan Game with Reinforcement Learning [121.93690719186412]
  Card game AI has always been a hot topic in the research of artificial intelligence. In this paper, we are devoted to developing an AI program for a more complex card game, GuanDan. We propose the first AI program DanZero for GuanDan using reinforcement learning technique.
  arXiv  Detail & Related papers  (2022-10-31T06:29:08Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)
• Suphx: Mastering Mahjong with Deep Reinforcement Learning [114.68233321904623]
  We design an AI for Mahjong, named Suphx, based on deep reinforcement learning with some newly introduced techniques. Suphx has demonstrated stronger performance than most top human players in terms of stable rank. This is the first time that a computer program outperforms most top human players in Mahjong.
  arXiv  Detail & Related papers  (2020-03-30T16:18:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.