Ranking a Set of Objects using Heterogeneous Workers: QUITE an Easy Problem

• URL: http://arxiv.org/abs/2310.02016v1
• Date: Tue, 3 Oct 2023 12:42:13 GMT
• Title: Ranking a Set of Objects using Heterogeneous Workers: QUITE an Easy Problem
• Authors: Alessandro Nordio and Alberto tarable and Emilio Leonardi
• Abstract summary: We focus on the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of unequal workers. We propose QUITE, a non-adaptive ranking algorithm that jointly estimates workers' reliabilities and qualities of objects.
• Score: 54.90613714264689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We focus on the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of unequal workers, each worker being characterized by a specific degree of reliability, which reflects her ability to rank pairs of objects. More specifically, we assume that objects are endowed with intrinsic qualities and that the probability with which an object is preferred to another depends both on the difference between the qualities of the two competitors and on the reliability of the worker. We propose QUITE, a non-adaptive ranking algorithm that jointly estimates workers' reliabilities and qualities of objects. Performance of QUITE is compared in different scenarios against previously proposed algorithms. Finally, we show how QUITE can be naturally made adaptive.

Related papers

• Rethinking Object Saliency Ranking: A Novel Whole-flow Processing Paradigm [22.038715439842044]
  This paper proposes a new paradigm for saliency ranking, which aims to completely focus on ranking salient objects by their "importance order" The proposed approach outperforms existing state-of-the-art methods on the widely-used SALICON set.
  arXiv  Detail & Related papers  (2023-12-06T01:51:03Z)
• SortNet: Learning To Rank By a Neural-Based Sorting Algorithm [5.485151775727742]
  We present SortNet, an adaptive ranking algorithm which orders objects using a neural network as a comparator. The proposed algorithm is evaluated on the LETOR dataset showing promising performances in comparison with other state of the art algorithms.
  arXiv  Detail & Related papers  (2023-11-03T12:14:26Z)
• Learning Embeddings for Sequential Tasks Using Population of Agents [27.61165606165948]
  We present an information-theoretic framework to learn fixed-dimensional embeddings for tasks in reinforcement learning. We leverage the idea that two tasks are similar if observing an agent's performance on one task reduces our uncertainty about its performance on the other.
  arXiv  Detail & Related papers  (2023-06-05T23:38:31Z)
• Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models [80.23791222509644]
  Inconsistent AI models are considered brittle and untrustworthy by human users. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks. We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
  arXiv  Detail & Related papers  (2023-03-28T16:57:12Z)
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective [93.56647950778357]
  Blind image quality assessment (BIQA) predicts the human perception of image quality without any reference information. We develop a general and automated multitask learning scheme for BIQA to exploit auxiliary knowledge from other tasks.
  arXiv  Detail & Related papers  (2023-03-27T07:58:09Z)
• A Case Study on Optimization of Warehouses [2.2101681534594237]
  In warehouses, order picking is known to be the most labor-intensive and costly task in which the employees account for a large part of the warehouse performance. In this work, we aim at optimizing the storage assignment and the order picking problem within mezzanine warehouse with regards to their reciprocal influence.
  arXiv  Detail & Related papers  (2021-11-23T07:22:57Z)
• Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons [85.5955376526419]
  In rank aggregation problems, users exhibit various accuracy levels when comparing pairs of items. We propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons. We prove that our algorithm can return the true ranking of items with high probability.
  arXiv  Detail & Related papers  (2021-10-08T13:51:55Z)
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning [59.62721526353915]
  Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities. Our method aims to leverage these commonalities by asking the question: What is the expected utility of each agent when only considering a randomly selected sub-group of its observed entities?''
  arXiv  Detail & Related papers  (2020-06-07T18:28:41Z)
• Ranking a set of objects: a graph based least-square approach [70.7866286425868]
  We consider the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of equal workers. We propose a class of non-adaptive ranking algorithms that rely on a least-squares intrinsic optimization criterion for the estimation of qualities.
  arXiv  Detail & Related papers  (2020-02-26T16:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.