Related papers: Measuring Intelligence Beyond Human Scale

Measuring Intelligence Beyond Human Scale

URL: http://arxiv.org/abs/2607.07040v1
Date: Wed, 08 Jul 2026 06:19:10 GMT
Title: Measuring Intelligence Beyond Human Scale
Abstract summary: How can we measure intelligence beyond human capability?<n>Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable.<n>We propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems.
Score: 10.592096146800555
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured. We describe practical protocols that reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. We instantiate the framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.

Related papers

Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
arXiv Detail & Related papers (2026-03-20T16:27:17Z)
From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making [0.5153774021264936]
Empirical evidence shows that many failures arise from miscalibrated reliance.<n>This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness.
arXiv Detail & Related papers (2026-03-19T13:35:29Z)
Measuring What AI Systems Might Do: Towards A Measurement Science in AI [19.687397197326817]
capabilities, propensities, skills, values, and abilities are routinely used interchangeably and conflated with observable performance.<n>We argue that capabilities and propensities are dispositional properties - stable features of systems characterised by counterfactual relationships.
arXiv Detail & Related papers (2026-02-10T12:59:41Z)
Variance-Bounded Evaluation of Entity-Centric AI Systems Without Ground Truth: Theory and Measurement [0.0]
We introduce VB-Score, a variance-bounded evaluation framework for entity-centric AI systems.<n> VB-Score enumerates plausible interpretations through constraint relaxation and Monte Carlo sampling.<n>It then evaluates system outputs by their expected success across interpretations, penalized by variance to assess robustness of the system.
arXiv Detail & Related papers (2025-09-26T07:54:38Z)
Evaluating Generative AI Systems is a Social Science Measurement Challenge [78.35388859345056]
We present a framework for measuring concepts related to the capabilities, impacts, opportunities, and risks of GenAI systems. The framework distinguishes between four levels: the background concept, the systematized concept, the measurement instrument(s), and the instance-level measurements themselves.
arXiv Detail & Related papers (2024-11-17T02:35:30Z)
Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
arXiv Detail & Related papers (2024-07-22T17:52:12Z)
Evaluating General-Purpose AI with Psychometrics [43.85432514910491]
We discuss the need for a comprehensive and accurate evaluation of general-purpose AI systems such as large language models. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation.
arXiv Detail & Related papers (2023-10-25T05:38:38Z)
Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
arXiv Detail & Related papers (2023-06-18T09:54:33Z)
Human Uncertainty in Concept-Based AI Systems [37.82747673914624]
We study human uncertainty in the context of concept-based AI systems. We show that training with uncertain concept labels may help mitigate weaknesses in concept-based systems.
arXiv Detail & Related papers (2023-03-22T19:17:57Z)
Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
arXiv Detail & Related papers (2022-01-27T22:15:56Z)
An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis [68.8204255655161]
We show that uncertainty measures based on Monte-Carlo dropout in the context of a human-in-the-loop system increase the system's transparency and performance. A simulation study demonstrates that the uncertainty-based human-in-the-loop system increases performance for different levels of human involvement.
arXiv Detail & Related papers (2020-07-14T15:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.