When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery

• URL: http://arxiv.org/abs/2507.01613v1
• Date: Wed, 02 Jul 2025 11:30:38 GMT
• Title: When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery
• Abstract summary: We propose a general parametric framework for modeling ordinal paired comparisons without ties.<n>We show that binarizing ordinal data can significantly improve the accuracy of ranking recovery.
• Score: 4.451479907610764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Paired comparison data, where users evaluate items in pairs, play a central role in ranking and preference learning tasks. While ordinal comparison data intuitively offer richer information than binary comparisons, this paper challenges that conventional wisdom. We propose a general parametric framework for modeling ordinal paired comparisons without ties. The model adopts a generalized additive structure, featuring a link function that quantifies the preference difference between two items and a pattern function that governs the distribution over ordinal response levels. This framework encompasses classical binary comparison models as special cases, by treating binary responses as binarized versions of ordinal data. Within this framework, we show that binarizing ordinal data can significantly improve the accuracy of ranking recovery. Specifically, we prove that under the counting algorithm, the ranking error associated with binary comparisons exhibits a faster exponential convergence rate than that of ordinal data. Furthermore, we characterize a substantial performance gap between binary and ordinal data in terms of a signal-to-noise ratio (SNR) determined by the pattern function. We identify the pattern function that minimizes the SNR and maximizes the benefit of binarization. Extensive simulations and a real application on the MovieLens dataset further corroborate our theoretical findings.

Related papers

• Pretraining on Call Graphs: When Binary Analysis Tasks Profit From Context [7.347989843033034]
  We conduct experiments using a range of graph-based models on the embeddings generated by two state-of-the-art binary function embedding models.<n>We find that optimizing for semantic similarity tasks correlates with worse performance on syntactic tasks.
  arXiv  Detail & Related papers  (2026-08-03T11:44:09Z)
• Learnable Weighting of Intra-Attribute Distances for Categorical Data Clustering with Nominal and Ordinal Attributes [60.91548718000255]
  We study the intrinsic difference and connection of nominal and ordinal attribute values from a perspective akin to the graph.<n>We propose a novel distance metric to measure the intra-attribute distances of nominal and ordinal attributes in a unified way.<n>We also propose a new clustering algorithm to make the learning of intra-attribute distance weights and partitions of data objects into a single learning paradigm.
  arXiv  Detail & Related papers  (2026-07-06T04:38:39Z)
• Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs [10.095945693317079]
  evaluating large language models (LLMs) is important for understanding their capabilities, comparing competing systems, and supporting the deployment of reliable models in practice.<n>For open-ended tasks, pairwise evaluation has become a popular paradigm, in which two responses to the same prompt are compared and the resulting judgments are aggregated into an overall ranking.<n>A central challenge of this paradigm is intransitivity: the induced comparison outcomes may fail to support any coherent global ranking.
  arXiv  Detail & Related papers  (2026-06-16T07:44:45Z)
• BinConv: A Neural Architecture for Ordinal Encoding in Time-Series Forecasting [5.827431686047649]
  We propose textbfBinConv, a fully convolutional neural network architecture designed for probabilistic forecasting.<n>BinConv achieves superior performance compared to widely used baseline datasets in both point and probabilistic forecasting.
  arXiv  Detail & Related papers  (2025-05-30T13:41:39Z)
• OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP [6.6002656593260225]
  Benchmark datasets play a crucial role in providing standardized testbeds that ensure fair comparisons.<n>Existing NLP ranking benchmarks typically use binary relevance labels or continuous relevance scores, neglecting ordinal relevance scores.<n>We introduce OrdRankBen, a novel benchmark designed to capture multi-granularity relevance distinctions.
  arXiv  Detail & Related papers  (2025-03-02T00:28:55Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• CORE: Learning Consistent Ordinal REpresentations for Image Ordinal Estimation [35.39143939072549]
  This paper proposes learning intrinsic Consistent Ordinal REpresentations (CORE) from ordinal relations residing in groundtruth labels. CORE can accurately construct an ordinal latent space and significantly enhance existing deep ordinal regression methods to achieve better results.
  arXiv  Detail & Related papers  (2023-01-15T15:42:26Z)
• Ordinal-ResLogit: Interpretable Deep Residual Neural Networks for Ordered Choices [6.982614422666432]
  We develop a fully interpretable deep learning-based ordinal regression model. Formulations for market share, substitution patterns, and elasticities are derived. Our results show that Ordinal-ResLogit outperforms the traditional ordinal regression model for both datasets.
  arXiv  Detail & Related papers  (2022-04-20T02:14:28Z)
• Riemannian classification of EEG signals with missing values [67.90148548467762]
  This paper proposes two strategies to handle missing data for the classification of electroencephalograms. The first approach estimates the covariance from imputed data with the $k$-nearest neighbors algorithm; the second relies on the observed data by leveraging the observed-data likelihood within an expectation-maximization algorithm. As results show, the proposed strategies perform better than the classification based on observed data and allow to keep a high accuracy even when the missing data ratio increases.
  arXiv  Detail & Related papers  (2021-10-19T14:24:50Z)
• Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons [85.5955376526419]
  In rank aggregation problems, users exhibit various accuracy levels when comparing pairs of items. We propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons. We prove that our algorithm can return the true ranking of items with high probability.
  arXiv  Detail & Related papers  (2021-10-08T13:51:55Z)
• Contrastive Fine-tuning Improves Robustness for Neural Rankers [1.3868793694964396]
  We present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. We use relevance labels to denote similar/dissimilar pairs, which allows the model to learn the underlying matching semantics across different query-document pairs. In experiments with four passage ranking datasets, the proposed contrastive fine-tuning method obtains improvements on robustness to query reformulations, noise perturbations, and zero-shot transfer for both BERT and BART based rankers.
  arXiv  Detail & Related papers  (2021-05-27T04:00:22Z)
• Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions [57.77347280992548]
  In this paper, we design two-sample tests for pairwise comparison data and ranking data. Our test requires essentially no assumptions on the distributions. By applying our two-sample test on real-world pairwise comparison data, we conclude that ratings and rankings provided by people are indeed distributed differently.
  arXiv  Detail & Related papers  (2020-06-21T20:51:09Z)
• Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator [62.26981903551382]
  Variational auto-encoders (VAEs) with binary latent variables provide state-of-the-art performance in terms of precision for document retrieval. We propose a pairwise loss function with discrete latent VAE to reward within-class similarity and between-class dissimilarity for supervised hashing. This new semantic hashing framework achieves superior performance compared to the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-05-21T06:11:33Z)
• Preference Modeling with Context-Dependent Salient Features [12.403492796441434]
  We consider the problem of estimating a ranking on a set of items from noisy pairwise comparisons given item features. Our key observation is that two items compared in isolation from other items may be compared based on only a salient subset of features.
  arXiv  Detail & Related papers  (2020-02-22T04:05:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.