A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding

• URL: http://arxiv.org/abs/2506.19083v2
• Date: Fri, 27 Jun 2025 12:02:46 GMT
• Title: A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding
• Authors: Alexander Goldberg, Giulia Fanti, Nihar B. Shah,
• Abstract summary: We propose a principled framework for randomized decision-making based on interval estimates of the quality of each item.<n>We introduce MERIT, an optimization-based method that maximizes the worst-case expected number of top candidates selected.<n>We prove that MERIT satisfies desirable axiomatic properties not guaranteed by existing approaches.
• Score: 68.43987626137512
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many decision-making processes involve evaluating and then selecting items; examples include scientific peer review, job hiring, school admissions, and investment decisions. The eventual selection is performed by applying rules or deliberations to the raw evaluations, and then deterministically selecting the items deemed to be the best. These domains feature error-prone evaluations and uncertainty about future outcomes, which undermine the reliability of such deterministic selection rules. As a result, selection mechanisms involving explicit randomization that incorporate the uncertainty are gaining traction in practice. However, current randomization approaches are ad hoc, and as we prove, inappropriate for their purported objectives. In this paper, we propose a principled framework for randomized decision-making based on interval estimates of the quality of each item. We introduce MERIT (Maximin Efficient Randomized Interval Top-k), an optimization-based method that maximizes the worst-case expected number of top candidates selected, under uncertainty represented by overlapping intervals (e.g., confidence intervals or min-max intervals). MERIT provides an optimal resource allocation scheme under an interpretable notion of robustness. We develop a polynomial-time algorithm to solve the optimization problem and demonstrate empirically that the method scales to over 10,000 items. We prove that MERIT satisfies desirable axiomatic properties not guaranteed by existing approaches. Finally, we empirically compare algorithms on synthetic peer review data. Our experiments demonstrate that MERIT matches the performance of existing algorithms in expected utility under fully probabilistic review data models used in previous work, while outperforming previous methods with respect to our novel worst-case formulation.

Related papers

• Conformal Information Pursuit for Interactively Guiding Large Language Models [64.39770942422288]
  This paper explores sequential querying strategies that aim to minimize the expected number of queries.<n>One such strategy is Information Pursuit (IP), a greedy algorithm that at each iteration selects the query that maximizes information gain or equivalently minimizes uncertainty.<n>We propose Conformal Information Pursuit (C-IP), an alternative approach to sequential information gain based on conformal prediction sets.
  arXiv  Detail & Related papers  (2025-07-04T03:55:39Z)
• Robust Information Selection for Hypothesis Testing with Misclassification Penalties [3.3444620077119436]
  We study the problem of robust information selection for a Bayesian hypothesis testing / classification task.<n>The goal is to identify the true state of the world from a finite set of hypotheses based on observations from selected information sources.<n>We introduce a novel misclassification penalty framework, which enables non-uniform treatment of different misclassification events.
  arXiv  Detail & Related papers  (2025-02-20T17:05:27Z)
• Sound Heuristic Search Value Iteration for Undiscounted POMDPs with Reachability Objectives [16.101435842520473]
  This paper studies the challenging yet important problem in POMDPs known as the (indefinite-horizon) Maximal Reachability Probability Problem. Inspired by the success of point-based methods developed for discounted problems, we study their extensions to MRPP. We present a novel algorithm that leverages the strengths of these techniques for efficient exploration of the belief space.
  arXiv  Detail & Related papers  (2024-06-05T02:33:50Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Online POMDP Planning with Anytime Deterministic Optimality Guarantees [9.444784653236157]
  We derive a deterministic relationship for discrete POMDPs between an approximated and the optimal solution.<n>We show that our derivations provide an avenue for a new set of algorithms and can be attached to existing algorithms.
  arXiv  Detail & Related papers  (2023-10-03T04:40:38Z)
• Selection by Prediction with Conformal p-values [7.917044695538599]
  We study screening procedures that aim to select candidates whose unobserved outcomes exceed user-specified values. We develop a method that wraps around any prediction model to produce a subset of candidates while controlling the proportion of falsely selected units.
  arXiv  Detail & Related papers  (2022-10-04T06:34:49Z)
• Generalizing Bayesian Optimization with Decision-theoretic Entropies [102.82152945324381]
  We consider a generalization of Shannon entropy from work in statistical decision theory. We first show that special cases of this entropy lead to popular acquisition functions used in BO procedures. We then show how alternative choices for the loss yield a flexible family of acquisition functions.
  arXiv  Detail & Related papers  (2022-10-04T04:43:58Z)
• Bi-objective Ranking and Selection Using Stochastic Kriging [0.0]
  We consider bi-objective ranking and selection problems in which the two objective outcomes have been observed with uncertainty. We propose a novel Bayesian bi-objective ranking and selection method that sequentially allocates extra samples to competitive solutions. Experimental results show that the proposed method outperforms the standard allocation method, as well as a well-known state-of-the-art algorithm.
  arXiv  Detail & Related papers  (2022-09-05T23:51:07Z)
• Off-Policy Evaluation with Policy-Dependent Optimization Response [90.28758112893054]
  We develop a new framework for off-policy evaluation with a textitpolicy-dependent linear optimization response. We construct unbiased estimators for the policy-dependent estimand by a perturbation method. We provide a general algorithm for optimizing causal interventions.
  arXiv  Detail & Related papers  (2022-02-25T20:25:37Z)
• On the Optimality of Batch Policy Optimization Algorithms [106.89498352537682]
  Batch policy optimization considers leveraging existing data for policy construction before interacting with an environment. We show that any confidence-adjusted index algorithm is minimax optimal, whether it be optimistic, pessimistic or neutral. We introduce a new weighted-minimax criterion that considers the inherent difficulty of optimal value prediction.
  arXiv  Detail & Related papers  (2021-04-06T05:23:20Z)
• Adaptive Sampling for Best Policy Identification in Markov Decision Processes [79.4957965474334]
  We investigate the problem of best-policy identification in discounted Markov Decision (MDPs) when the learner has access to a generative model. The advantages of state-of-the-art algorithms are discussed and illustrated.
  arXiv  Detail & Related papers  (2020-09-28T15:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.