When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

• URL: http://arxiv.org/abs/2603.13688v1
• Date: Sat, 14 Mar 2026 01:47:38 GMT
• Title: When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions
• Abstract summary: We propose a general decision-theoretic framework for human-AI collaboration.<n>We treat AI assessments as factor-level signals and human judgments as costly information that can be selectively acquired.<n>We apply our framework to AI-assisted peer review.
• Score: 2.7755032786791003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems increasingly assist human decision making by producing preliminary assessments of complex inputs. However, such AI-generated assessments can often be noisy or systematically biased, raising a central question: how should costly human effort be allocated to correct AI outputs where it matters the most for the final decision? We propose a general decision-theoretic framework for human-AI collaboration in which AI assessments are treated as factor-level signals and human judgments as costly information that can be selectively acquired. We consider cases where the optimal selection problem reduces to maximizing a reward associated with each candidate subset of factors, and turn policy design into reward estimation. We develop estimation procedures under both nonparametric and linear models, covering contextual and non-contextual selection rules. In the linear setting, the optimal rule admits a closed-form expression with a clear interpretation in terms of factor importance and residual variance. We apply our framework to AI-assisted peer review. Our approach substantially outperforms LLM-only predictions and achieves performance comparable to full human review while using only 20-30% of the human information. Across different selection rules, we find that simpler rules derived under linear models can significantly reduce computational cost without harming final prediction performance. Our results highlight both the value of human intervention and the efficiency of principled dispatching.

Related papers

• Robust Human-AI Complementarity under Uncertainty [16.894647212197782]
  We study how asymmetric information about the quality of information available to a human decision maker vs. an AI impacts the ability of a decision maker to extract complementary value from AI predictions.<n>We show that a key factor is the error correlation structure between human and AI predictions.<n>In particular, when the AI's prediction errors are textitnegatively correlated with those of the human, the decision maker can construct robust strategies which guarantee improvements in expected utility.
  arXiv  Detail & Related papers  (2026-07-07T17:58:06Z)
• Cascaded Language Models for Cost-effective Human-AI Decision-Making [52.81324217423194]
  We present a cascaded LLM decision framework that adaptively delegates tasks across multiple tiers of expertise.<n>First, a deferral policy determines whether to accept the base model's answer or regenerate it with a large model.<n>Second, an abstention policy decides whether the cascade model response is sufficiently certain or requires human intervention.
  arXiv  Detail & Related papers  (2025-06-13T15:36:22Z)
• Human aversion? Do AI Agents Judge Identity More Harshly Than Performance [0.06554326244334868]
  We investigate how AI agents based on large language models assess and integrate human input.<n>We find that the AI system systematically discounts human advice, penalizing human errors more severely than algorithmic errors.
  arXiv  Detail & Related papers  (2025-03-31T02:05:27Z)
• Aligning Large Language Models from Self-Reference AI Feedback with one General Principle [61.105703857868775]
  We propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference. Finally, we determine which answer better fits human preferences according to the criticism.
  arXiv  Detail & Related papers  (2024-06-17T03:51:46Z)
• Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies [0.43981305860983716]
  We show how to compare the performance of three alternative decision-making systems--human-alone, human-with-AI, and AI-alone. We find that the risk assessment recommendations do not improve the classification accuracy of a judge's decision to impose cash bail.
  arXiv  Detail & Related papers  (2024-03-18T01:04:52Z)
• A Decision Theoretic Framework for Measuring AI Reliance [20.669176502049066]
  Humans frequently make decisions with the aid of artificially intelligent (AI) systems.<n>Researchers have identified ensuring that a human has appropriate reliance on an AI as a critical component of achieving complementary performance.<n>We propose a formal definition of reliance, based on statistical decision theory, which separates the concepts of reliance as the probability the decision-maker follows the AI's recommendation.
  arXiv  Detail & Related papers  (2024-01-27T09:13:09Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism [91.52263068880484]
  We study offline Reinforcement Learning with Human Feedback (RLHF) We aim to learn the human's underlying reward and the MDP's optimal policy from a set of trajectories induced by human choices. RLHF is challenging for multiple reasons: large state space but limited human feedback, the bounded rationality of human decisions, and the off-policy distribution shift.
  arXiv  Detail & Related papers  (2023-05-29T01:18:39Z)
• Learning Complementary Policies for Human-AI Teams [13.371050441794651]
  This paper tackles the challenge of human-AI complementarity in decision-making.<n>We develop a robust solution for human-AI collaboration when outcomes are only observed under assigned actions.<n>We show that substantial performance improvements are achievable by routing only a small fraction of instances to human decision-makers.
  arXiv  Detail & Related papers  (2023-02-06T17:22:18Z)
• Off-Policy Evaluation with Policy-Dependent Optimization Response [90.28758112893054]
  We develop a new framework for off-policy evaluation with a textitpolicy-dependent linear optimization response. We construct unbiased estimators for the policy-dependent estimand by a perturbation method. We provide a general algorithm for optimizing causal interventions.
  arXiv  Detail & Related papers  (2022-02-25T20:25:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.