RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue

• URL: http://arxiv.org/abs/2606.13310v1
• Date: Thu, 11 Jun 2026 13:07:02 GMT
• Title: RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue
• Abstract summary: We present RogueAI, an interactive webapp that operationalizes this test as a one-on-two interrogation game.<n>The player's task is to identify the deceptive agent and "shut it off" before a turn budget is exhausted.<n>We introduce AutoRogueAI, a procedural extension in which players co-design a custom scenario with a narrator agent that secretly chooses its own deception strategy.
• Score: 2.7606655162305476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The original Turing Test asks a human judge to distinguish a machine from a person through dialogue. Three quarters of a century later, conversational systems pass this test in casual settings; the interesting epistemological question has shifted. We argue that the relevant modern variant asks not whether a dialogue partner is artificial, but whether it can be trusted. We present RogueAI, an interactive webapp that operationalizes this revisited test as a one-on-two interrogation game: a human player questions two indistinguishable Large Language Model agents, knowing that exactly one of them has been licensed to deceive within a shared fictional scenario. The player's task is to identify the deceptive agent and "shut it off" before a turn budget is exhausted. We further introduce AutoRogueAI, a procedural extension in which players co-design a custom scenario with a narrator agent that secretly chooses its own deception strategy. We describe the framing, sketch the abstract architecture and gameplay loop, and situate the artifact within recent work on LLM deception, social-deduction benchmarks, and scalable oversight via debate. A three-day pilot deployment (467 initiated sessions, 415 completed, 1876 interaction turns in Italian) provides early feasibility evidence and surfaces a concrete tension: the deceptive agent carries a reliable, locally-present linguistic signature - differential helpfulness, brevity, hedging - that a simple heuristic exploits at 75.6% accuracy, yet human players achieved only 56.6%, consistent with ignoring the most diagnostic signal entirely. We discuss what this gap implies for the artifact's use as a data-collection vehicle, a teaching tool, and an evaluation harness for honesty-trained models.

Related papers

• A Contextual-Bandit Oversight Game with Two-Sided Informational Asymmetry [0.3384279376065155]
  We study runtime human oversight of an AI agent when private information runs in both directions.<n>This is the kind of asymmetry that arises naturally when an autonomous robot or software agent has inspected a situation its human supervisor cannot directly assess.<n>We introduce a contextual-bandit team game with two-sided asymmetric information and a play/ask/trust/oversee interface.
  arXiv  Detail & Related papers  (2026-06-30T20:30:01Z)
• Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue [1.1305136905804842]
  We present Inverse Turing Bench, a benchmark that evaluates models on their ability to differentiate humans and AI in multi-turn text.<n>The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI.<n>We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy.
  arXiv  Detail & Related papers  (2026-06-20T02:47:56Z)
• RealityTest: How People Probe AI Identity and Whether Models Disclose It [5.4667629504496285]
  We present RealityTest to test whether AI systems disclose their identity when asked.<n>We release the underlying dataset of 3,152 identity-probing queries collected from 750 participants across 49 countries and five languages.<n>We find that only 31% of people ask about identity directly in ambiguous scenarios, and that the questions people ask are far more diverse than machine-generated queries.
  arXiv  Detail & Related papers  (2026-05-29T12:40:16Z)
• ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search [49.61286310968402]
  We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem.<n>An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses.<n>The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who), spatial reasoning (Where), and temporal reasoning (When)
  arXiv  Detail & Related papers  (2026-04-14T14:06:19Z)
• Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models [75.85319609088354]
  Sentient Agent as a Judge (SAGE) is an evaluation framework for large language models.<n>SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction.<n>SAGE provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.
  arXiv  Detail & Related papers  (2025-05-01T19:06:10Z)
• Conversation Games and a Strategic View of the Turing Test [0.0]
  We focus on a subset of the games, called verdict games.<n>In a verdict game, two players alternate to contribute to a conversation, which is evaluated at each stage by a non-strategic judge.<n>We show the practical relevance of the proposed concepts by simulation experiments, and show that a strategic agent outperforms a naive agent by a high margin.
  arXiv  Detail & Related papers  (2025-01-30T16:08:37Z)
• Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues [54.81155589931697]
  Collaborative Instance object Navigation (CoIN) is a new task setting where the agent actively resolve uncertainties about the target instance.<n>We propose a novel training-free method, Agent-user Interaction with UncerTainty Awareness (AIUTA)<n>First, upon object detection, a Self-Questioner model initiates a self-dialogue within the agent to obtain a complete and accurate observation description.<n>An Interaction Trigger module determines whether to ask a question to the human, continue or halt navigation.
  arXiv  Detail & Related papers  (2024-12-02T08:16:38Z)
• X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents [56.64615470513102]
  The Turing test examines whether AIs exhibit human-like behaviour in natural language conversations.<n>Traditional setting limits each participant to one message at a time and requires constant human participation.<n>This paper proposes textbftextscX-Turing, which enhances the original test with a textitburst dialogue pattern.
  arXiv  Detail & Related papers  (2024-08-19T09:57:28Z)
• Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systems [64.10696852552103]
  Highly anthropomorphic responses might make users uncomfortable or implicitly deceive them into thinking they are interacting with a human. We collect human ratings on the feasibility of approximately 900 two-turn dialogs sampled from 9 diverse data sources.
  arXiv  Detail & Related papers  (2022-10-22T12:10:44Z)
• Partially Fake Audio Detection by Self-attention-based Fake Span Discovery [89.21979663248007]
  We propose a novel framework by introducing the question-answering (fake span discovery) strategy with the self-attention mechanism to detect partially fake audios. Our submission ranked second in the partially fake audio detection track of ADD 2022.
  arXiv  Detail & Related papers  (2022-02-14T13:20:55Z)
• "Wait, I'm Still Talking!" Predicting the Dialogue Interaction Behavior Using Imagine-Then-Arbitrate Model [24.560203199376478]
  In real human-human conversations, human often sequentially sends several short messages for readability instead of a long message in one turn. We propose a novel Imagine-then-Arbitrate (ITA) neural dialogue model to help the agent decide whether to wait or to make a response directly.
  arXiv  Detail & Related papers  (2020-02-22T04:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.