Distinct citation distributions complicate research evaluations. A single indicator that universally reveals research efficiency cannot be formulated

• URL: http://arxiv.org/abs/2407.09138v1
• Date: Fri, 12 Jul 2024 10:16:21 GMT
• Title: Distinct citation distributions complicate research evaluations. A single indicator that universally reveals research efficiency cannot be formulated
• Authors: Alonso Rodríguez-Navarro,
• Abstract summary: Size-independent, top percentile-based indicators are accurate when the global ranks of local publications fit a power law. deviations in the least cited papers are frequent in countries and occur in all journals with high impact factors.
• Score: 0.0
• License:
• Abstract: Purpose: Analyze the diversity of citation distributions to publications in different research topics to investigate the accuracy of size-independent, rank-based indicators. Top percentile-based indicators are the most common indicators of this type, and the evaluations of Japan are the most evident misjudgments. Design/methodology/approach: The distributions of citations to publications from countries and in journals in several research topics were analyzed along with the corresponding global publications using histograms with logarithmic binning, double rank plots, and normal probability plots of log-transformed numbers of citations. Findings: Size-independent, top percentile-based indicators are accurate when the global ranks of local publications fit a power law, but deviations in the least cited papers are frequent in countries and occur in all journals with high impact factors. In these cases, a single indicator is misleading. Comparisons of proportions of uncited papers are the best way to predict these deviations. Research limitations: The study is fundamentally analytical; its results describe mathematical facts that are self-evident. Practical implications: Respectable institutions, such as the OECD, European Commission, US National Science Board, and others, produce research country rankings and individual evaluations using size-independent percentile indicators that are misleading in many countries. These misleading evaluations should be discontinued because they cause confusion among research policymakers and lead to incorrect research policies. Originality/value: Studies linking the lower tail of citation distribution, including uncited papers, to percentile research indicators have not been performed previously. The present results demonstrate that studies of this type are necessary to find reliable procedures for research assessments.

Related papers

• Biases in gendered citation practices: an exploratory study and some reflections on the Matthew and Matilda effects [2.0277446818410994]
  This paper aims at analyzing gendered citation practices in the software engineering literature. Our results show that some efforts still need to be done to achieve fairness in citation practices in the SE field.
  arXiv  Detail & Related papers  (2024-09-18T19:53:41Z)
• Comprehensive Equity Index (CEI): Definition and Application to Bias Evaluation in Biometrics [47.762333925222926]
  We present a novel metric to quantify biased behaviors of machine learning models. We focus on and apply it to the operational evaluation of face recognition systems.
  arXiv  Detail & Related papers  (2024-09-03T14:19:38Z)
• Measuring and Addressing Indexical Bias in Information Retrieval [69.7897730778898]
  PAIR framework supports automatic bias audits for ranked documents or entire IR systems. After introducing DUO, we run an extensive evaluation of 8 IR systems on a new corpus of 32k synthetic and 4.7k natural documents. A human behavioral study validates our approach, showing that our bias metric can help predict when and how indexical bias will shift a reader's opinion.
  arXiv  Detail & Related papers  (2024-06-06T17:42:37Z)
• Semantic Properties of cosine based bias scores for word embeddings [48.0753688775574]
  We propose requirements for bias scores to be considered meaningful for quantifying biases. We analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
  arXiv  Detail & Related papers  (2024-01-27T20:31:10Z)
• The Impact of Differential Feature Under-reporting on Algorithmic Fairness [86.275300739926]
  We present an analytically tractable model of differential feature under-reporting. We then use to characterize the impact of this kind of data bias on algorithmic fairness. Our results show that, in real world data settings, under-reporting typically leads to increasing disparities.
  arXiv  Detail & Related papers  (2024-01-16T19:16:22Z)
• Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition [4.336779198334903]
  One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models.
  arXiv  Detail & Related papers  (2023-03-28T11:04:18Z)
• Geographic Citation Gaps in NLP Research [63.13508571014673]
  This work asks a series of questions on the relationship between geographical location and publication success. We first created a dataset of 70,000 papers from the ACL Anthology, extracted their meta-information, and generated their citation network. We show that not only are there substantial geographical disparities in paper acceptance and citation but also that these disparities persist even when controlling for a number of variables such as venue of publication and sub-field of NLP.
  arXiv  Detail & Related papers  (2022-10-26T02:25:23Z)
• The SAME score: Improved cosine based bias score for word embeddings [49.75878234192369]
  We introduce SAME, a novel bias score for semantic bias in embeddings. We show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature [4.339510167603376]
  The goal of this study is to characterize the reporting of empiricism in Computing Education Research (CER) literature. We conducted an SLR of 427 papers published during 2014 and 2015 in five CER venues. Over 80% of papers had some form of empirical evaluation.
  arXiv  Detail & Related papers  (2021-07-02T16:37:29Z)
• How are journals cited? characterizing journal citations by type of citation [0.0]
  We present initial results on the statistical characterization of citations to journals based on citation function. We also present initial results of characterizing the ratio of supports and disputes received by a journal as a potential indicator of quality.
  arXiv  Detail & Related papers  (2021-02-22T14:15:50Z)
• The Demise of Single-Authored Publications in Computer Science: A Citation Network Analysis [0.0]
  I analyze the DBLP database to study role of single author publications in the computer science literature between 1940 and 2019. I examine the demographics and reception by computing the population fraction, citation statistics, and scores of single author publications over the years.
  arXiv  Detail & Related papers  (2020-01-02T07:47:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.