When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery

• URL: http://arxiv.org/abs/2507.01613v1
• Date: Wed, 02 Jul 2025 11:30:38 GMT
• Title: When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery
• Authors: Shirong Xu, Jingnan Zhang, Junhui Wang,
• Abstract summary: We propose a general parametric framework for modeling ordinal paired comparisons without ties.<n>We show that binarizing ordinal data can significantly improve the accuracy of ranking recovery.
• Score: 4.451479907610764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Paired comparison data, where users evaluate items in pairs, play a central role in ranking and preference learning tasks. While ordinal comparison data intuitively offer richer information than binary comparisons, this paper challenges that conventional wisdom. We propose a general parametric framework for modeling ordinal paired comparisons without ties. The model adopts a generalized additive structure, featuring a link function that quantifies the preference difference between two items and a pattern function that governs the distribution over ordinal response levels. This framework encompasses classical binary comparison models as special cases, by treating binary responses as binarized versions of ordinal data. Within this framework, we show that binarizing ordinal data can significantly improve the accuracy of ranking recovery. Specifically, we prove that under the counting algorithm, the ranking error associated with binary comparisons exhibits a faster exponential convergence rate than that of ordinal data. Furthermore, we characterize a substantial performance gap between binary and ordinal data in terms of a signal-to-noise ratio (SNR) determined by the pattern function. We identify the pattern function that minimizes the SNR and maximizes the benefit of binarization. Extensive simulations and a real application on the MovieLens dataset further corroborate our theoretical findings.

Related papers

• OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP [6.6002656593260225]
  Benchmark datasets play a crucial role in providing standardized testbeds that ensure fair comparisons.<n>Existing NLP ranking benchmarks typically use binary relevance labels or continuous relevance scores, neglecting ordinal relevance scores.<n>We introduce OrdRankBen, a novel benchmark designed to capture multi-granularity relevance distinctions.
  arXiv  Detail & Related papers  (2025-03-02T00:28:55Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• CORE: Learning Consistent Ordinal REpresentations for Image Ordinal Estimation [35.39143939072549]
  This paper proposes learning intrinsic Consistent Ordinal REpresentations (CORE) from ordinal relations residing in groundtruth labels. CORE can accurately construct an ordinal latent space and significantly enhance existing deep ordinal regression methods to achieve better results.
  arXiv  Detail & Related papers  (2023-01-15T15:42:26Z)
• Ordinal-ResLogit: Interpretable Deep Residual Neural Networks for Ordered Choices [6.982614422666432]
  We develop a fully interpretable deep learning-based ordinal regression model. Formulations for market share, substitution patterns, and elasticities are derived. Our results show that Ordinal-ResLogit outperforms the traditional ordinal regression model for both datasets.
  arXiv  Detail & Related papers  (2022-04-20T02:14:28Z)
• Riemannian classification of EEG signals with missing values [67.90148548467762]
  This paper proposes two strategies to handle missing data for the classification of electroencephalograms. The first approach estimates the covariance from imputed data with the $k$-nearest neighbors algorithm; the second relies on the observed data by leveraging the observed-data likelihood within an expectation-maximization algorithm. As results show, the proposed strategies perform better than the classification based on observed data and allow to keep a high accuracy even when the missing data ratio increases.
  arXiv  Detail & Related papers  (2021-10-19T14:24:50Z)
• Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons [85.5955376526419]
  In rank aggregation problems, users exhibit various accuracy levels when comparing pairs of items. We propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons. We prove that our algorithm can return the true ranking of items with high probability.
  arXiv  Detail & Related papers  (2021-10-08T13:51:55Z)
• Contrastive Fine-tuning Improves Robustness for Neural Rankers [1.3868793694964396]
  We present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. We use relevance labels to denote similar/dissimilar pairs, which allows the model to learn the underlying matching semantics across different query-document pairs. In experiments with four passage ranking datasets, the proposed contrastive fine-tuning method obtains improvements on robustness to query reformulations, noise perturbations, and zero-shot transfer for both BERT and BART based rankers.
  arXiv  Detail & Related papers  (2021-05-27T04:00:22Z)
• Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions [57.77347280992548]
  In this paper, we design two-sample tests for pairwise comparison data and ranking data. Our test requires essentially no assumptions on the distributions. By applying our two-sample test on real-world pairwise comparison data, we conclude that ratings and rankings provided by people are indeed distributed differently.
  arXiv  Detail & Related papers  (2020-06-21T20:51:09Z)
• Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator [62.26981903551382]
  Variational auto-encoders (VAEs) with binary latent variables provide state-of-the-art performance in terms of precision for document retrieval. We propose a pairwise loss function with discrete latent VAE to reward within-class similarity and between-class dissimilarity for supervised hashing. This new semantic hashing framework achieves superior performance compared to the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-05-21T06:11:33Z)
• Preference Modeling with Context-Dependent Salient Features [12.403492796441434]
  We consider the problem of estimating a ranking on a set of items from noisy pairwise comparisons given item features. Our key observation is that two items compared in isolation from other items may be compared based on only a salient subset of features.
  arXiv  Detail & Related papers  (2020-02-22T04:05:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.