SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction

• URL: http://arxiv.org/abs/2604.17141v2
• Date: Tue, 21 Apr 2026 07:55:19 GMT
• Title: SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction
• Abstract summary: SciImpact is a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields.<n>It comprises 215,928 contrastive paper pairs reflecting meaningful impact differences in both short-term and long-term settings.<n>We evaluate 11 widely used large language models (LLMs) on SciImpact.
• Score: 10.526822187607658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid growth of scientific literature calls for automated methods to assess and predict research impact. Prior work has largely focused on citation-based metrics, leaving limited evaluation of models' capability to reason about other impact dimensions. To this end, we introduce SciImpact, a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields. SciImpact captures various forms of scientific influence, ranging from citation counts to award recognition, media attention, patent reference, and artifact adoption, by integrating heterogeneous data sources and targeted web crawling. It comprises 215,928 contrastive paper pairs reflecting meaningful impact differences in both short-term (e.g., Best Paper Award) and long-term settings (e.g., Nobel Prize). We evaluate 11 widely used large language models (LLMs) on SciImpact. Results show that off-the-shelf models exhibit substantial variability across dimensions and fields, while multi-task supervised fine-tuning consistently enables smaller LLMs (e.g., 4B) to markedly outperform much larger models (e.g., 30B) and surpass powerful closed-source LLMs (e.g., o4-mini). These results establish SciImpact as a challenging benchmark and demonstrate its value for multi-dimensional, multi-field scientific impact prediction. Our project homepage is https://flypig23.github.io/sciimpact-homepage/

Related papers

• AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era [55.37558486468993]
  We introduce AgentIdeaBench, a benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration.<n>We report matched Static-Active evaluations for 33 LLMs across 40 densely scored subfields spanning five disciplines.<n>Active exploration reveals considerably more capability headroom, and that headroom is unevenly distributed across models.
  arXiv  Detail & Related papers  (2026-09-07T15:19:13Z)
• Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery [41.32793603340132]
  Large language models (LLMs) are increasingly involved in scientific discovery.<n>It remains unclear whether they can support complex real laboratory science.<n> Evaluation of 19 large language models (MLLMs) shows that even the best-performing model reaches only 48.7% accuracy.
  arXiv  Detail & Related papers  (2026-08-07T08:06:31Z)
• LLM-Metrics: Measuring Research Impact Through Large Language Model Memory [0.09685837672183746]
  We propose LLM-Metrics, a research-impact assessment metric derived from the parametric memory of large language models (LLMs)<n>We evaluated 549 computer science papers published in 2023-2024 across 17 LLMs spanning 0.5B to 72B parameters from six vendors.<n>Of the 17 models, 15 produced positive predictions, 9 of which were significant at p less than 0.05, with an overall Spearman correlation of rho = 0.1495 and p = 0.0004 against citation counts.
  arXiv  Detail & Related papers  (2026-05-21T08:45:57Z)
• Evaluating Large Language Models in Scientific Discovery [91.732562776782]
  Large language models (LLMs) are increasingly applied to scientific research, yet prevailing science benchmarks probe decontextualized knowledge.<n>We introduce a scenario-grounded benchmark that evaluates LLMs across biology, chemistry, materials, and physics.<n>The framework assesses models at two levels: (i) question-level accuracy on scenario-tied items and (ii) project-level performance.
  arXiv  Detail & Related papers  (2025-12-17T16:20:03Z)
• Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, LLaMA [0.6263481844384227]
  This work proposes a systematic evaluation framework to examine how interaction tone affects model accuracy.<n>We apply this framework to three recently released and widely available large language models: GPT-4o mini (OpenAI), Gemini 2.0 Flash (Google DeepMind), and Llama 4 Scout (Meta)<n>Our results show that tone sensitivity is both model-dependent and domain-specific. Neutral or Very Friendly prompts generally yield higher accuracy than Very Rude prompts, but statistically significant effects appear only in a subset of Humanities tasks.
  arXiv  Detail & Related papers  (2025-12-14T19:25:20Z)
• Optimizing Research Portfolio For Semantic Impact [55.2480439325792]
  Citation metrics are widely used to assess academic impact but suffer from social biases.<n>We introduce rXiv Semantic Impact (XSI), a novel framework that predicts research impact.<n>XSI tracks the evolution of research concepts in the academic knowledge graph.
  arXiv  Detail & Related papers  (2025-02-19T17:44:13Z)
• From Words to Worth: Newborn Article Impact Prediction with LLM [69.41680520058418]
  This paper introduces a promising approach, leveraging the capabilities of LLMs to predict the future impact of newborn articles.<n>The proposed method employs LLM to discern the shared semantic features of highly impactful papers from a large collection of title-abstract pairs.<n>The quantitative results, with an MAE of 0.216 and an NDCG@20 of 0.901, demonstrate that the proposed approach achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-08-07T17:52:02Z)
• Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
  This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations.<n>We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.<n>Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
  arXiv  Detail & Related papers  (2024-05-27T13:26:34Z)
• Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
  We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
  arXiv  Detail & Related papers  (2024-04-01T17:45:15Z)
• Time to Cite: Modeling Citation Networks using the Dynamic Impact Single-Event Embedding Model [0.33123773366516646]
  citation networks are a prominent example of single-event dynamic networks. We propose a novel likelihood function for the characterization of such single-event networks. The Dynamic Impact Single-Event Embedding model (DISEE) reconciles static latent distance network embedding approaches with classical dynamic impact assessments.
  arXiv  Detail & Related papers  (2024-02-28T22:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.