Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue

• URL: http://arxiv.org/abs/2606.21844v1
• Date: Sat, 20 Jun 2026 02:47:56 GMT
• Title: Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue
• Abstract summary: We present Inverse Turing Bench, a benchmark that evaluates models on their ability to differentiate humans and AI in multi-turn text.<n>The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI.<n>We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy.
• Score: 1.1305136905804842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI systems integrate into online spaces, differentiating them from humans in conversations is increasingly important. We present Inverse Turing Bench, a benchmark that evaluates LLMs and other models on their ability to differentiate humans and AI in multi-turn text. The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI. The task is to correctly identify which dialogue is human-only vs. human-AI. We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy: 89.41%, 77.92%, and 75.94% respectively. Our results suggest that statistical approaches to detection have semantic blind spots, but semantic approaches are susceptible to persona-prompting. Our work speaks to the Inverse Turing Test as a probe of LLM theory of mind, and motivates human-AI differentiation as a critical capability for AI systems. Our live benchmark can be found at https://huggingface.co/spaces/roc-hci/Inverse-Turing-Bench-Leaderboard (anonymity preserved).

Related papers

• Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction [32.28977425466535]
  We conduct the first Turing test for S2S systems, collecting 2,968 human judgments on dialogues between 9 state-of-the-art S2S systems and 28 human participants.<n>No existing evaluated S2S system passes the test, revealing a significant gap in human-likeness.<n>We develop a fine-grained taxonomy of 18 human-likeness dimensions and crowd-annotate our collected dialogues accordingly.
  arXiv  Detail & Related papers  (2026-02-27T15:15:31Z)
• LVLMs and Humans Ground Differently in Referential Communication [33.82075906105276]
  We present a factorial design involving director-matcher pairs that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels.<n>We release the online pipeline for data collection, the tools and analyses for accuracy, efficiency, and lexical overlap, and a corpus of 356 dialogues that unmasks LVLMs' limitations in interactively resolving referring expressions.
  arXiv  Detail & Related papers  (2026-01-27T16:52:20Z)
• The next question after Turing's question: Introducing the Grow-AI test [51.56484100374058]
  This study aims to extend the framework for assessing artificial intelligence, called GROW-AI.<n>GROW-AI is designed to answer the question "Can machines grow up?" -- a natural successor to the Turing Test.<n>The originality of the work lies in the conceptual transposition of the process of "growing" from the human world to that of artificial intelligence.
  arXiv  Detail & Related papers  (2025-08-22T10:19:42Z)
• X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents [56.64615470513102]
  The Turing test examines whether AIs exhibit human-like behaviour in natural language conversations.<n>Traditional setting limits each participant to one message at a time and requires constant human participation.<n>This paper proposes textbftextscX-Turing, which enhances the original test with a textitburst dialogue pattern.
  arXiv  Detail & Related papers  (2024-08-19T09:57:28Z)
• GPT-4 is judged more human than humans in displaced and inverted Turing tests [0.7437224586066946]
  Everyday AI detection requires differentiating between people and AI in online conversations. We measured how well people and large language models can discriminate using two modified versions of the Turing test: inverted and displaced.
  arXiv  Detail & Related papers  (2024-07-11T20:28:24Z)
• How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO [55.25989137825992]
  We introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. We evaluate three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models.
  arXiv  Detail & Related papers  (2024-04-22T08:00:51Z)
• BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues [72.65163468440434]
  This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting. We prompt large language models (LLMs) to generate a full multi-turn dialogue based on the ChatSEED, utterance by utterance. We find GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts.
  arXiv  Detail & Related papers  (2023-10-20T16:53:51Z)
• Evaluating Human-Language Model Interaction [79.33022878034627]
  We develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems. We design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. We find that better non-interactive performance does not always translate to better human-LM interaction.
  arXiv  Detail & Related papers  (2022-12-19T18:59:45Z)
• Can Machines Imitate Humans? Integrative Turing-like tests for Language and Vision Demonstrate a Narrowing Gap [56.611702960809644]
  We benchmark AI's ability to imitate humans in three language tasks and three vision tasks.<n>Next, we conducted 72,191 Turing-like tests with 1,916 human judges and 10 AI judges.<n>Imitation ability showed minimal correlation with conventional AI performance metrics.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)
• Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurs [22.52332536886295]
  We present a novel formulation of the interaction between the human and the AI as a sequential game. We show that in this case the AI's problem of helping bounded-rational humans make better decisions reduces to a Bayes-adaptive POMDP. We discuss ways in which the machine can learn to improve upon its own limitations as well with the help of the human.
  arXiv  Detail & Related papers  (2022-04-03T21:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.