Related papers: Generative AI and the future of scientometrics: current topics and future questions

Generative AI and the future of scientometrics: current topics and future questions

URL: http://arxiv.org/abs/2507.00783v1
Date: Tue, 01 Jul 2025 14:22:16 GMT
Title: Generative AI and the future of scientometrics: current topics and future questions
Authors: Benedetto Lepori, Jens Peter Andersen, Karsten Donnay,
Abstract summary: The aim of this paper is to review the use of GenAI in scientometrics, and to begin a debate on the broader implications for the field.<n>We provide an introduction on GenAI's generative and probabilistic nature as rooted in distributional linguistics.<n>We relate this to the debate on the extent to which GenAI might be able to mimic human'reasoning'
Score: 0.1638581561083717
License: http://creativecommons.org/licenses/by/4.0/
Abstract: The aim of this paper is to review the use of GenAI in scientometrics, and to begin a debate on the broader implications for the field. First, we provide an introduction on GenAI's generative and probabilistic nature as rooted in distributional linguistics. And we relate this to the debate on the extent to which GenAI might be able to mimic human 'reasoning'. Second, we leverage this distinction for a critical engagement with recent experiments using GenAI in scientometrics, including topic labelling, the analysis of citation contexts, predictive applications, scholars' profiling, and research assessment. GenAI shows promise in tasks where language generation dominates, such as labelling, but faces limitations in tasks that require stable semantics, pragmatic reasoning, or structured domain knowledge. However, these results might become quickly outdated. Our recommendation is, therefore, to always strive to systematically compare the performance of different GenAI models for specific tasks. Third, we inquire whether, by generating large amounts of scientific language, GenAI might have a fundamental impact on our field by affecting textual characteristics used to measure science, such as authors, words, and references. We argue that careful empirical work and theoretical reflection will be essential to remain capable of interpreting the evolving patterns of knowledge production.

Related papers

Generative AI in Science: Applications, Challenges, and Emerging Questions [0.49157446832511503]
The review draws on the OpenAlex publication database to identify scientific literature related to GenAI.<n>The analysis finds that while there is a rapid adoption of GenAI in science and science practice, its long-term implications remain unclear.<n>The study provides early insights into GenAI's growing role in science and identifies questions for future research in this evolving field.
arXiv Detail & Related papers (2025-07-11T05:02:24Z)
Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis [20.57872238271025]
We conduct a meta-analysis to evaluate the effect of GenAI on the performance in creative tasks.<n>Our results show no significant difference in creative performance between GenAI and humans.<n>GenAI has a significant negative effect on the diversity of ideas for such collaborations between humans and GenAI.
arXiv Detail & Related papers (2025-05-22T19:39:10Z)
GenAI vs. Human Fact-Checkers: Accurate Ratings, Flawed Rationales [2.3475022003300055]
GPT-4o, one of the most used AI models in consumer applications, outperforms other models, but all models exhibit only moderate agreement with human coders.<n>We also assess the effectiveness of summarized versus full content inputs, finding that summarized content holds promise for improving efficiency without sacrificing accuracy.
arXiv Detail & Related papers (2025-02-20T17:47:40Z)
Computational Safety for Generative AI: A Signal Processing Perspective [65.268245109828]
computational safety is a mathematical framework that enables the quantitative assessment, formulation, and study of safety challenges in GenAI.<n>We show how sensitivity analysis and loss landscape analysis can be used to detect malicious prompts with jailbreak attempts.<n>We discuss key open research challenges, opportunities, and the essential role of signal processing in computational AI safety.
arXiv Detail & Related papers (2025-02-18T02:26:50Z)
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge [78.35388859345056]
We argue that the ML community would benefit from learning from and drawing on the social sciences when developing measurement instruments for evaluating GenAI systems.<n>We present a four-level framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, behaviors, and impacts of GenAI systems.
arXiv Detail & Related papers (2025-02-01T21:09:51Z)
"So what if I used GenAI?" -- Implications of Using Cloud-based GenAI in Software Engineering Research [0.0]
This paper sheds light on the various research aspects in which GenAI is used, thus raising awareness of its legal implications to novice and budding researchers.<n>We summarize key aspects regarding our current knowledge that every software researcher involved in using GenAI should be aware of to avoid critical mistakes that may expose them to liability claims.
arXiv Detail & Related papers (2024-12-10T06:18:15Z)
Dimensions of Generative AI Evaluation Design [51.541816010127256]
We propose a set of general dimensions that capture critical choices involved in GenAI evaluation design. These dimensions include the evaluation setting, the task type, the input source, the interaction style, the duration, the metric type, and the scoring method.
arXiv Detail & Related papers (2024-11-19T18:25:30Z)
Model-based Maintenance and Evolution with GenAI: A Look into the Future [47.93555901495955]
We argue that Generative Artificial Intelligence (GenAI) can be used as a means to address the limitations of Model-Based Engineering (MBM&E) We propose that GenAI can be used in MBM&E for: reducing engineers' learning curve, maximizing efficiency with recommendations, or serving as a reasoning tool to understand domain problems.
arXiv Detail & Related papers (2024-07-09T23:13:26Z)
A Systematic Review of Generative AI for Teaching and Learning Practice [0.37282630026096586]
There are no agreed guidelines towards the usage of GenAI systems in higher education. There is a need for additional interdisciplinary, multidimensional studies in HE through collaboration.
arXiv Detail & Related papers (2024-06-13T18:16:27Z)
Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda [1.8592384822257952]
We elaborate on why XAI has gained importance with the rise of GenAI and its challenges for explainability research. We also unveil novel and emerging desiderata that explanations should fulfill, covering aspects such as verifiability, interactivity, security, and cost.
arXiv Detail & Related papers (2024-04-15T08:18:16Z)
Identifying and Mitigating the Security Risks of Generative AI [179.2384121957896]
This paper reports the findings of a workshop held at Google on the dual-use dilemma posed by GenAI. GenAI can be used just as well by attackers to generate new attacks and increase the velocity and efficacy of existing attacks. We discuss short-term and long-term goals for the community on this topic.
arXiv Detail & Related papers (2023-08-28T18:51:09Z)
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI [65.44737844681256]
A lack of consensus on how to evaluate explainable AI (XAI) hinders the advancement of the field. We argue that one way to close the gap is to develop evaluation methods that account for different user requirements.
arXiv Detail & Related papers (2022-06-22T05:17:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.