(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable

• URL: http://arxiv.org/abs/2606.12848v1
• Date: Thu, 11 Jun 2026 03:22:24 GMT
• Title: (Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable
• Abstract summary: We argue that the reliability of AI-assisted research depends not only on model capability, but also on how cognitive labour is structured between humans and machines.<n>We study this problem through Human-in-the-Loop Economic Research (HLER), a decision architecture based on pre-commitment, decision sequencing, accountability, and attention allocation.
• Score: 15.442870614597451
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used for tasks once reserved for trained researchers, including hypothesis generation, specification choice, and drafting conclusions. We argue that the reliability of AI-assisted research depends not only on model capability, but also on how cognitive labour is structured between humans and machines. We study this problem through Human-in-the-Loop Economic Research (HLER), a decision architecture based on pre-commitment, decision sequencing, accountability, and attention allocation. In a pre-specified 2*4 factorial experiment with 280 complete research runs across four datasets, an unconstrained multi-agent baseline produced critical failures in 72% of runs. Using the same underlying model, the same agent decomposition, and identical prompts for the shared reasoning agents, HLER reduced the failure rate to 16% by imposing three architectural commitments: LLMs reason but do not execute data work, data and estimation are handled deterministically, and three human decision gates bind the workflow. Fisher's exact test rejects equality of failure rates at p<0.001. Reliability gains were largest on the least publicly represented dataset, a Qing-dynasty population register, consistent with a task-based production model with Frechet-distributed output quality. An 80-run ablation suggests that deterministic computation and human gates contribute independently, with exploratory evidence of complementarity. We interpret HLER as a research harness rather than an autonomous AI scientist: it sharply reduces failures, makes residual weaknesses more visible, and prevents unreliable claims from being advanced as publication-ready outputs.

Related papers

• CausalGame: Benchmarking Causal Thinking of LLM Agents in Games [55.189910630332065]
  CausalGame is a benchmark for evaluating the causal thinking capabilities of AI Scientists.<n>We design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders.<n>The best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics.
  arXiv  Detail & Related papers  (2026-07-05T13:08:51Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents [75.01735520608075]
  Existing benchmarks mostly test literal Theory of Mind (ToM) by asking direct belief questions.<n>We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household.<n>On the hard split, all seven evaluated frontier models score 0.0% Pass3 on functional task completion, while averaging 45.0% on literal belief probes.
  arXiv  Detail & Related papers  (2026-05-11T00:04:19Z)
• AI scientists produce results without reasoning scientifically [3.100302590436282]
  Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously.<n>Here, we evaluate LLM-based scientific agents across eight domains, spanning execution to hypothesis-driven inquiry.
  arXiv  Detail & Related papers  (2026-04-20T20:23:42Z)
• Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys [0.0]
  Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions.<n>We study the design problem of allocating a fixed budget of human respondents across estimation tasks when cheap LLM predictions are available for every task.<n>Our framework combines three components. First, building on Prediction-Powered Inference, we characterize a question-specific rectification difficulty that governs how quickly the estimator's variance decreases with human sample size.<n>Second, we derive a closed-form optimal allocation rule that directs more human labels to tasks where the LLM is least reliable.
  arXiv  Detail & Related papers  (2026-04-19T05:45:37Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• From Experiments to Expertise: Scientific Knowledge Consolidation for AI-Driven Computational Research [3.7193230342956056]
  We present QMatSuite, an open-source platform for AI-driven computational science.<n>QMatSuite stores findings with provenance, retrieves knowledge before new calculations, and in dedicated reflection sessions correct erroneous findings.<n>In benchmarks on a six-step quantum-mechanical simulation workflow, accumulated knowledge reduces reasoning overhead by 67% and improves accuracy from 47% to 3% deviation from literature.
  arXiv  Detail & Related papers  (2026-03-13T17:25:47Z)
• HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery [15.29303145822579]
  We introduce HLER (Human-in-the-Loop Economic Research), a multi-agent architecture that supports empirical research automation.<n>The system orchestrates specialized agents for data auditing, data profiling, hypothesis generation, econometric analysis, manuscript drafting, and automated review.<n>Experiments show dataset-aware hypothesis generation produces feasible research questions in 87% of cases, while complete empirical manuscripts can be produced at an average API cost of $0.8-$1.5 per run.
  arXiv  Detail & Related papers  (2026-03-08T03:40:34Z)
• On the Influence of Artificial Intelligence on Human Problem-Solving: Empirical Insights for the Third Wave in a Multinational Longitudinal Pilot Study [0.0]
  This article investigates the evolving paradigm of human-AI collaboration in problem-solving contexts.<n>Building upon previous waves, our findings reveal the consolidation of a hybrid problem-solving culture.<n>The study concludes that educational and technological interventions must prioritize verification scaffolds.
  arXiv  Detail & Related papers  (2025-11-13T10:20:07Z)
• Cascaded Language Models for Cost-effective Human-AI Decision-Making [52.81324217423194]
  We present a cascaded LLM decision framework that adaptively delegates tasks across multiple tiers of expertise.<n>First, a deferral policy determines whether to accept the base model's answer or regenerate it with a large model.<n>Second, an abstention policy decides whether the cascade model response is sufficiently certain or requires human intervention.
  arXiv  Detail & Related papers  (2025-06-13T15:36:22Z)
• Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
  Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
  arXiv  Detail & Related papers  (2022-01-27T22:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.