Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

• URL: http://arxiv.org/abs/2607.02467v1
• Date: Thu, 02 Jul 2026 17:34:37 GMT
• Title: Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting
• Abstract summary: This pilot shows that the value of human-AI collaboration depends on a specific, measurable form of human capital.<n>The results are preliminary but statistically robust, and motivate a pre-registered replication now in preparation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Whether pairing people with AI helps or hurts is usually reported as a single average effect. Using a real-money prediction market (Polymarket) as an objective, externally resolved benchmark, this pilot shows that the value of human-AI collaboration depends on a specific, measurable form of human capital. Analyzed at the level of the individual forecaster, hybrid performance is trimodal: most people either deferred to the model (matching it) or used it to rubber-stamp a prior guess (performing worse than the model alone), while a minority engaged in genuine complementary reasoning and reached accuracy matching or even exceeding (i.e., lower error than) the market itself. Collaborative traits (perspective-taking, intellectual humility, and curiosity) rather than raw cognitive ability or model benchmarks, distinguished who reached that mode. The results are preliminary but statistically robust, and motivate a pre-registered replication now in preparation.

Related papers

• Dynamically Allocating Evaluation Effort for Model Ranking [60.00410111594536]
  We formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms.<n>We show that it improves discrimination between top-performing models.<n>This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
  arXiv  Detail & Related papers  (2026-08-04T10:33:04Z)
• (Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable [15.442870614597451]
  We argue that the reliability of AI-assisted research depends not only on model capability, but also on how cognitive labour is structured between humans and machines.<n>We study this problem through Human-in-the-Loop Economic Research (HLER), a decision architecture based on pre-commitment, decision sequencing, accountability, and attention allocation.
  arXiv  Detail & Related papers  (2026-06-11T03:22:24Z)
• Empirically evaluating commonsense intelligence in large language models with large-scale human judgments [4.212429064310439]
  We propose a method for evaluating common sense in artificial intelligence.<n>We measure the correspondence between a model's judgment and that of a human population.<n>Our framework contributes to the growing call for adapting AI models to human collectivities that possess different, often incompatible, social stocks of knowledge.
  arXiv  Detail & Related papers  (2025-05-15T13:55:27Z)
• More than Marketing? On the Information Value of AI Benchmarks for Practitioners [42.73526862595375]
  In academia, public benchmarks were generally viewed as suitable measures for capturing research progress.<n>In product and policy, benchmarks were often found to be inadequate for informing substantive decisions.<n>We conclude that effective benchmarks should provide meaningful, real-world evaluations, incorporate domain expertise, and maintain transparency in scope and goals.
  arXiv  Detail & Related papers  (2024-12-07T03:35:39Z)
• Generative Models, Humans, Predictive Models: Who Is Worse at High-Stakes Decision Making? [10.225573060836478]
  Large generative models (LMs) are already being used for decision making tasks that were previously done by predictive models or humans.<n>We put popular LMs to the test in a high-stakes decision making task: recidivism prediction.
  arXiv  Detail & Related papers  (2024-10-20T19:00:59Z)
• Building Socially-Equitable Public Models [32.35090986784889]
  Public models offer predictions to a variety of downstream tasks and have played a crucial role in various AI applications. We advocate for integrating the objectives of downstream agents into the optimization process. We propose a novel Equitable Objective to address performance disparities and foster fairness among heterogeneous agents in training.
  arXiv  Detail & Related papers  (2024-06-04T21:27:43Z)
• Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition [99.7047087527422]
  In this work, we demonstrate that competition can fundamentally alter the behavior of machine learning scaling trends. We find many settings where improving data representation quality decreases the overall predictive accuracy across users. At a conceptual level, our work suggests that favorable scaling trends for individual model-providers need not translate to downstream improvements in social welfare.
  arXiv  Detail & Related papers  (2023-06-26T13:06:34Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study [65.17429512679695]
  In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models.
  arXiv  Detail & Related papers  (2021-06-02T00:48:33Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.