The Generalized Turing Test: A Foundation for Comparing Intelligence

• URL: http://arxiv.org/abs/2605.10851v1
• Date: Mon, 11 May 2026 17:00:18 GMT
• Title: The Generalized Turing Test: A Foundation for Comparing Intelligence
• Abstract summary: We introduce a formal framework for comparing the capabilities of arbitrary agents via indistinguishability.<n>For agents A and B, we define the Turing comparator A $geq$ B to hold if B, acting as a distinguisher, cannot reliably distinguish between interactions with A and another instance of B.<n>This yields a dataset- and task-agnostic notion of relative intelligence.
• Score: 0.8929189891250133
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We introduce the Generalized Turing Test (GTT), a formal framework for comparing the capabilities of arbitrary agents via indistinguishability. For agents A and B, we define the Turing comparator A $\geq$ B to hold if B, acting as a distinguisher, cannot reliably distinguish between interactions with A (instructed to imitate B) and another instance of B. This yields a dataset- and task-agnostic notion of relative intelligence. We study the comparator's structure, including conditions under which it is transitive and therefore induces an ordering over equivalence classes, and we define and analyze variants with querying, bounded interaction, and fixed distinguishers. To complement the theory, we instantiate the framework on a collection of modern models, empirically evaluating pairwise indistinguishability across thousands of trials. The resulting comparisons exhibit a stratified structure consistent with existing rankings, hinting that the proposed framework yields meaningful empirical orderings. Our results position indistinguishability as a unifying lens for reasoning about intelligence, suggesting a foundation for evaluation and, potentially, training objectives that are inherently independent of fixed datasets or benchmarks.

Related papers

• Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations [12.535445487099393]
  We present a human-grounded evaluation framework that quantifies alignment between SAE latents and human-annotated concepts.<n>We validate this matching through targeted attribute perturbations.<n>Our framework suggests that moderate dictionary sizes provide the best trade-off, yielding the most interpretable SAEs.
  arXiv  Detail & Related papers  (2026-06-23T15:39:29Z)
• Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents [81.79110139097297]
  General-purpose embedding models excel at recognizing semantic similarities but fail to capture the characteristics of texts specified by user instructions.<n>We reframe instruction-following clustering as a generative task and train large reasoning models as autonomous clustering agents.<n>Our reasoning-driven training pipeline enables LRMs to interpret high-level clustering instructions and then infer the corresponding latent groupings.
  arXiv  Detail & Related papers  (2026-03-06T10:15:54Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering [97.52852990265136]
  We introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models.<n>We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-21T07:35:36Z)
• Grouping Nodes With Known Value Differences: A Lossless UCT-based Abstraction Algorithm [64.43268969806098]
  A core challenge of Monte Carlo Tree Search (MCTS) is its sample efficiency, which can be improved by grouping state-action pairs.<n>We break with the paradigm of grouping value-equivalent states or state-action pairs and instead group states and state-action pairs with possibly different values.<n>We call this abstraction framework Known Value Difference Abstractions ( KVDA), which infers the value differences by analysis of the immediate rewards.
  arXiv  Detail & Related papers  (2025-10-29T11:03:44Z)
• ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity [9.337443482551356]
  We introduce ASSESS (A Semantic and Structural Evaluation Framework for Statement Similarity), which comprehensively integrates semantic and structural information to provide a continuous similarity score.<n>For rigorous validation, we present EPLA, a new benchmark of 524 expert-annotated formal statement pairs derived from miniF2F and ProofNet.<n>Experiments on EPLA demonstrate that TransTED Similarity outperforms existing methods, achieving state-of-the-art accuracy and the highest Kappa coefficient.
  arXiv  Detail & Related papers  (2025-09-26T12:02:58Z)
• Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference [1.2309843977641421]
  We introduce a multi-rationale explainable object recognition benchmark comprising datasets in which each image is annotated with multiple ground-truth rationales.<n>We propose a contrastive conditional inference framework that explicitly models the probabilistic relationships among image embeddings, category labels, and rationales.<n>Our approach achieves state-of-the-art results on the multi-rationale explainable object recognition benchmark, including strong zero-shot performance.
  arXiv  Detail & Related papers  (2025-08-19T21:28:12Z)
• Bhatt Conjectures: On Necessary-But-Not-Sufficient Benchmark Tautology for Human Like Reasoning [0.0]
  Bhatt Conjectures framework introduces rigorous, hierarchical benchmarks for evaluating AI reasoning and understanding.<n>Agentreasoning-sdk demonstrates practical implementation, revealing that current AI models struggle with complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-06-13T02:41:18Z)
• Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation [24.081573908824353]
  First-order logic (FOL) reasoning is pivotal for intelligent systems.<n>Existing benchmarks often rely on extensive human annotation or handcrafted templates.<n>We propose a novel framework called ProverGen that synergizes the generative strengths of Large Language Models with the rigor and precision of symbolic provers.
  arXiv  Detail & Related papers  (2025-02-10T15:31:54Z)
• StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs [78.84060166851805]
  StructTest is a novel benchmark that evaluates large language models (LLMs) on their ability to follow compositional instructions and generate structured outputs.<n> Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets.<n>We demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o.
  arXiv  Detail & Related papers  (2024-12-23T22:08:40Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Machine Reading Comprehension using Case-based Reasoning [92.51061570746077]
  We present an accurate and interpretable method for answer extraction in machine reading comprehension. Our method builds upon the hypothesis that contextualized answers to similar questions share semantic similarities with each other.
  arXiv  Detail & Related papers  (2023-05-24T07:09:56Z)
• Aggregating Pairwise Semantic Differences for Few-Shot Claim Veracity Classification [21.842139093124512]
  We introduce SEED, a novel vector-based method to claim veracity classification. We build on the hypothesis that we can simulate class representative vectors that capture average semantic differences for claim-evidence pairs in a class. Experiments conducted on the FEVER and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings.
  arXiv  Detail & Related papers  (2022-05-11T17:23:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.