How Reliable are University Rankings?

• URL: http://arxiv.org/abs/2004.09006v1
• Date: Mon, 20 Apr 2020 01:00:59 GMT
• Title: How Reliable are University Rankings?
• Authors: Ali Dasdan, Eric Van Lare, and Bosko Zivaljevic
• Abstract summary: We take a fresh look at this ranking scheme using the public College dataset. We show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative. We conclude by making the case that all data and methods used for rankings should be made open for validation and repeatability.
• Score: 0.7646713951724009
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: University or college rankings have almost become an industry of their own, published by US News \& World Report (USNWR) and similar organizations. Most of the rankings use a similar scheme: Rank universities in decreasing score order, where each score is computed using a set of attributes and their weights; the attributes can be objective or subjective while the weights are always subjective. This scheme is general enough to be applied to ranking objects other than universities. As shown in the related work, these rankings have important implications and also many issues. In this paper, we take a fresh look at this ranking scheme using the public College dataset; we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed. For example, we show how to derive reasonable weights programmatically to move multiple universities in our dataset to the top rank; moreover, this task takes a few seconds for over 600 universities on a personal laptop. Our mathematical formulation, methods, and results are applicable to ranking objects other than universities too. We conclude by making the case that all the data and methods used for rankings should be made open for validation and repeatability.

Related papers

• AGRaME: Any-Granularity Ranking with Multi-Vector Embeddings [53.78802457488845]
  We introduce the idea of any-granularity ranking, which leverages multi-vector embeddings to rank at varying levels of granularity. We demonstrate the application of proposition-level ranking to post-hoc citation addition in retrieval-augmented generation.
  arXiv  Detail & Related papers  (2024-05-23T20:04:54Z)
• Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models [63.714662435555674]
  Large language models (LLMs) exhibit positional bias in how they use context. We propose permutation self-consistency, a form of self-consistency over ranking list outputs of black-box LLMs. Our approach improves scores from conventional inference by up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B)
  arXiv  Detail & Related papers  (2023-10-11T17:59:02Z)
• TRIVEA: Transparent Ranking Interpretation using Visual Explanation of Black-Box Algorithmic Rankers [4.336037935247747]
  Ranking schemes drive many real-world decisions, like, where to study, whom to hire, what to buy, etc. At the heart of most of these decisions are opaque ranking schemes, which dictate the ordering of data entities. We aim to enable transparency in ranking interpretation by using algorithmic rankers that learn from available data and by enabling human reasoning about the learned ranking differences using explainable AI (XAI) methods.
  arXiv  Detail & Related papers  (2023-08-28T16:58:44Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Learning List-Level Domain-Invariant Representations for Ranking [59.3544317373004]
  We propose list-level alignment -- learning domain-invariant representations at the higher level of lists. The benefits are twofold: it leads to the first domain adaptation generalization bound for ranking, in turn providing theoretical support for the proposed method.
  arXiv  Detail & Related papers  (2022-12-21T04:49:55Z)
• Weight Set Decomposition for Weighted Rank Aggregation: An interpretable and visual decision support tool [0.0]
  We show the wealth information that is available for the weighted rank aggregation problem due to its structure. We apply weight set decomposition to the set of convex properties useful for decomposition, and visualize the indifference regions.
  arXiv  Detail & Related papers  (2022-05-31T19:09:55Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)
• On the Linear Ordering Problem and the Rankability of Data [0.0]
  We use the degree of linearity to quantify what percentage of the data aligns with an optimal ranking. In a sports context, this is analogous to the number of games that a ranking can correctly predict in hindsight. We show that the optimal rankings computed via the LOP maximize the hindsight accuracy of a ranking.
  arXiv  Detail & Related papers  (2021-04-12T21:05:17Z)
• Measuring University Impact: Wikipedia approach [1.7894318579694353]
  We discuss the new methodological technique that evaluates the impact of university based on popularity of their alumni's pages on Wikipedia. Preliminary analysis shows that the number of page-views is higher for the contemporary persons that prove the perspectives of this approach. The ranking based on the alumni popularity was compared with the ranking of universities based on the popularity of their webpages on Wikipedia.
  arXiv  Detail & Related papers  (2020-12-27T17:41:56Z)
• PiRank: Learning To Rank via Differentiable Sorting [85.28916333414145]
  We propose PiRank, a new class of differentiable surrogates for ranking. We show that PiRank exactly recovers the desired metrics in the limit of zero temperature.
  arXiv  Detail & Related papers  (2020-12-12T05:07:36Z)
• Controlling Fairness and Bias in Dynamic Learning-to-Rank [31.41843594914603]
  We propose a learning algorithm that ensures notions of amortized group fairness, while simultaneously learning the ranking function from implicit feedback data. The algorithm takes the form of a controller that integrates unbiased estimators for both fairness and utility. In addition to its rigorous theoretical foundation and convergence guarantees, we find empirically that the algorithm is highly practical and robust.
  arXiv  Detail & Related papers  (2020-05-29T17:57:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.