Uncertain research country rankings. Should we continue producing uncertain rankings?

• URL: http://arxiv.org/abs/2312.17560v3
• Date: Thu, 20 Mar 2025 14:41:55 GMT
• Title: Uncertain research country rankings. Should we continue producing uncertain rankings?
• Authors: Alonso Rodriguez-Navarro,
• Abstract summary: Citation-based assessments of countries' research capabilities often misrepresent their ability to achieve breakthrough advancements.<n>The study evaluates the effectiveness of top-percentile citation metrics as indicators of breakthrough research.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Purpose: Citation-based assessments of countries' research capabilities often misrepresent their ability to achieve breakthrough advancements. These assessments commonly classify Japan as a developing country, which contradicts its prominent scientific standing. The purpose of this study is to investigate the underlying causes of such inaccurate assessments and to propose methods for conducting more reliable evaluations. Design/methodology/approach: The study evaluates the effectiveness of top-percentile citation metrics as indicators of breakthrough research. Using case studies of selected countries and research topics, the study examines how deviations from lognormal citation distributions impact the accuracy of these percentile indicators. A similar analysis is conducted using university data from the Leiden Ranking to investigate citation distribution deviations at the institutional level. Findings: The study finds that inflated lower tails in citation distributions lead to undervaluation of research capabilities in advanced technological countries, as captured by some percentile indicators. Conversely, research-intensive universities exhibit the opposite trend: a reduced lower tail relative to the upper tail, which causes percentile indicators to overestimate their actual research capacity. Research limitations: The descriptions are mathematical facts that are self-evident. Practical implications: Due to variations in citation patterns across countries and institutions, the Ptop 10%/P and Ptop 1%/P ratios are not universal predictors of breakthrough research. Evaluations should move away from these metrics. Relying on inappropriate citation-based measures could lead to poor decision-making in research policy, undermining the effectiveness of research strategies and their outcomes.

Related papers

• Biases in gendered citation practices: an exploratory study and some reflections on the Matthew and Matilda effects [2.0277446818410994]
  This paper aims at analyzing gendered citation practices in the software engineering literature.<n>Our results show that some efforts still need to be done to achieve fairness in citation practices in the SE field.
  arXiv  Detail & Related papers  (2024-09-18T19:53:41Z)
• Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! [80.3129093617928]
  Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics. This work highlights two issues with the meta-evaluation framework currently employed in WMT, and assesses their impact on the metrics rankings. We introduce the concept of sentinel metrics, which are designed explicitly to scrutinize the meta-evaluation process's accuracy, robustness, and fairness.
  arXiv  Detail & Related papers  (2024-08-25T13:29:34Z)
• Distinct citation distributions complicate research evaluations. A single indicator that universally reveals research efficiency cannot be formulated [0.0]
  Size-independent, top percentile-based indicators are accurate when the global ranks of local publications fit a power law. deviations in the least cited papers are frequent in countries and occur in all journals with high impact factors.
  arXiv  Detail & Related papers  (2024-07-12T10:16:21Z)
• Challenges and Considerations in the Evaluation of Bayesian Causal Discovery [49.0053848090947]
  Representing uncertainty in causal discovery is a crucial component for experimental design, and more broadly, for safe and reliable causal decision making. Unlike non-Bayesian causal discovery, which relies on a single estimated causal graph and model parameters for assessment, causal discovery presents challenges due to the nature of its quantity. No consensus on the most suitable metric for evaluation.
  arXiv  Detail & Related papers  (2024-06-05T12:45:23Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [55.33653554387953]
  Pattern Analysis and Machine Intelligence (PAMI) has led to numerous literature reviews aimed at collecting and fragmented information.<n>This paper presents a thorough analysis of these literature reviews within the PAMI field.<n>We try to address three core research questions: (1) What are the prevalent structural and statistical characteristics of PAMI literature reviews; (2) What strategies can researchers employ to efficiently navigate the growing corpus of reviews; and (3) What are the advantages and limitations of AI-generated reviews compared to human-authored ones.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Semantic Properties of cosine based bias scores for word embeddings [48.0753688775574]
  We propose requirements for bias scores to be considered meaningful for quantifying biases. We analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
  arXiv  Detail & Related papers  (2024-01-27T20:31:10Z)
• Predicting Survey Response with Quotation-based Modeling: A Case Study on Favorability towards the United States [0.0]
  We propose a pioneering approach for predicting survey responses by examining quotations using machine learning. We leverage a vast corpus of quotations from individuals across different nationalities to extract their level of favorability. We employ a combination of natural language processing techniques and machine learning algorithms to construct a predictive model for survey responses.
  arXiv  Detail & Related papers  (2023-05-23T14:11:01Z)
• Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias [33.99768156365231]
  We introduce a causal formulation for bias measurement in generative language models. We propose a benchmark called OccuGender, with a bias-measuring procedure to investigate occupational gender bias. The results show that these models exhibit substantial occupational gender bias.
  arXiv  Detail & Related papers  (2022-12-20T22:41:24Z)
• On predicting research grants productivity [0.38073142980732994]
  We analyzed whether bibliometric features are able to predict the success of research grants. Research subject and publication history play a role in predicting productivity. While the best results outperformed text-based attributes, the evaluated features were not highly discriminative.
  arXiv  Detail & Related papers  (2021-06-20T14:21:46Z)
• A Measure of Research Taste [91.3755431537592]
  We present a citation-based measure that rewards both productivity and taste. The presented measure, CAP, balances the impact of publications and their quantity. We analyze the characteristics of CAP for highly-cited researchers in biology, computer science, economics, and physics.
  arXiv  Detail & Related papers  (2021-05-17T18:01:47Z)
• Machine Learning and Data Science approach towards trend and predictors analysis of CDC Mortality Data for the USA [0.0]
  The study concluded (based on a sample) life expectancy regardless of gender, and their central tendencies; Marital status of the people also affected how frequent deaths were for each of them. The study shows that machine learning predictions aren't as viable for the data as it might be apparent.
  arXiv  Detail & Related papers  (2020-09-11T12:46:57Z)
• A Survey on Causal Inference [64.45536158710014]
  Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics. Various causal effect estimation methods for observational data have sprung up.
  arXiv  Detail & Related papers  (2020-02-05T21:35:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.