Bounded-Abstention Pairwise Learning to Rank

• URL: http://arxiv.org/abs/2505.23437v1
• Date: Thu, 29 May 2025 13:35:39 GMT
• Title: Bounded-Abstention Pairwise Learning to Rank
• Authors: Antonio Ferrara, Andrea Pugnana, Francesco Bonchi, Salvatore Ruggieri,
• Abstract summary: Abstention enables algorithmic decision-making system to defer uncertain or low-confidence decisions to human experts.<n>We introduce a novel method for abstention in pairwise learning-to-rank tasks.<n>Our contributions are threefold: a theoretical characterization of the optimal abstention strategy, a model-agnostic, plug-in algorithm for constructing abstaining ranking models, and a comprehensive empirical evaluations across multiple datasets.
• Score: 21.876570823233656
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ranking systems influence decision-making in high-stakes domains like health, education, and employment, where they can have substantial economic and social impacts. This makes the integration of safety mechanisms essential. One such mechanism is $\textit{abstention}$, which enables algorithmic decision-making system to defer uncertain or low-confidence decisions to human experts. While abstention have been predominantly explored in the context of classification tasks, its application to other machine learning paradigms remains underexplored. In this paper, we introduce a novel method for abstention in pairwise learning-to-rank tasks. Our approach is based on thresholding the ranker's conditional risk: the system abstains from making a decision when the estimated risk exceeds a predefined threshold. Our contributions are threefold: a theoretical characterization of the optimal abstention strategy, a model-agnostic, plug-in algorithm for constructing abstaining ranking models, and a comprehensive empirical evaluations across multiple datasets, demonstrating the effectiveness of our approach.

Related papers

• Towards a Cascaded LLM Framework for Cost-effective Human-AI Decision-Making [55.2480439325792]
  We present a cascaded LLM decision framework that adaptively delegates tasks across multiple tiers of expertise.<n>First, a deferral policy determines whether to accept the base model's answer or regenerate it with the large model.<n>Second, an abstention policy decides whether the cascade model response is sufficiently certain or requires human intervention.
  arXiv  Detail & Related papers  (2025-06-13T15:36:22Z)
• Risk-aware Classification via Uncertainty Quantification [9.641001762056876]
  We introduce three foundational desiderata for developing real-world risk-aware classification systems.<n>We demonstrate the unity between these principles and Evidential Deep Learning's operational attributes.<n>We then augment EDL empowering autonomous agents to exercise discretion during structured decision-making when uncertainty and risks are inherent.
  arXiv  Detail & Related papers  (2024-12-04T15:20:12Z)
• Sequential Manipulation Against Rank Aggregation: Theory and Algorithm [119.57122943187086]
  We leverage an online attack on the vulnerable data collection process. From the game-theoretic perspective, the confrontation scenario is formulated as a distributionally robust game. The proposed method manipulates the results of rank aggregation methods in a sequential manner.
  arXiv  Detail & Related papers  (2024-07-02T03:31:21Z)
• Beyond Expectations: Learning with Stochastic Dominance Made Practical [88.06211893690964]
  dominance models risk-averse preferences for decision making with uncertain outcomes. Despite theoretically appealing, the application of dominance in machine learning has been scarce. We first generalize the dominance concept to enable feasible comparisons between any arbitrary pair of random variables. We then develop a simple and efficient approach for finding the optimal solution in terms of dominance.
  arXiv  Detail & Related papers  (2024-02-05T03:21:23Z)
• Risk-Sensitive Stochastic Optimal Control as Rao-Blackwellized Markovian Score Climbing [3.9410617513331863]
  optimal control of dynamical systems is a crucial challenge in sequential decision-making. Control-as-inference approaches have had considerable success, providing a viable risk-sensitive framework to address the exploration-exploitation dilemma. This paper introduces a novel perspective by framing risk-sensitive control as Markovian reinforcement score climbing under samples drawn from a conditional particle filter.
  arXiv  Detail & Related papers  (2023-12-21T16:34:03Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• On the Complexity of Adversarial Decision Making [101.14158787665252]
  We show that the Decision-Estimation Coefficient is necessary and sufficient to obtain low regret for adversarial decision making. We provide new structural results that connect the Decision-Estimation Coefficient to variants of other well-known complexity measures.
  arXiv  Detail & Related papers  (2022-06-27T06:20:37Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)
• Morshed: Guiding Behavioral Decision-Makers towards Better Security Investment in Interdependent Systems [10.960507931439317]
  We model the behavioral biases of human decision-making in securing interdependent systems. We show that such behavioral decision-making leads to a suboptimal pattern of resource allocation. We propose three learning techniques for enhancing decision-making in multi-round setups.
  arXiv  Detail & Related papers  (2020-11-12T18:23:55Z)
• Fair Meta-Learning For Few-Shot Classification [7.672769260569742]
  A machine learning algorithm trained on biased data tends to make unfair predictions. We propose a novel fair fast-adapted few-shot meta-learning approach that efficiently mitigates biases during meta-train. We empirically demonstrate that our proposed approach efficiently mitigates biases on model output and generalizes both accuracy and fairness to unseen tasks.
  arXiv  Detail & Related papers  (2020-09-23T22:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.