SciEvent: Benchmarking Multi-domain Scientific Event Extraction

• URL: http://arxiv.org/abs/2509.15620v1
• Date: Fri, 19 Sep 2025 05:32:50 GMT
• Title: SciEvent: Benchmarking Multi-domain Scientific Event Extraction
• Authors: Bofu Dong, Pritesh Shah, Sumedh Sonawane, Tiyasha Banerjee, Erin Brady, Xinya Du, Ming Jiang,
• Abstract summary: We introduce SciEvent, a novel multi-domain benchmark of scientific abstracts annotated via a unified event extraction (EE) schema.<n>It includes 500 abstracts across five research domains, with manual annotations of event segments, triggers, and fine-grained arguments.<n>Experiments with fine-tuned EE models, large language models (LLMs), and human annotators reveal a performance gap, with current models struggling in domains such as sociology and humanities.
• Score: 14.37001604445613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific information extraction (SciIE) has primarily relied on entity-relation extraction in narrow domains, limiting its applicability to interdisciplinary research and struggling to capture the necessary context of scientific information, often resulting in fragmented or conflicting statements. In this paper, we introduce SciEvent, a novel multi-domain benchmark of scientific abstracts annotated via a unified event extraction (EE) schema designed to enable structured and context-aware understanding of scientific content. It includes 500 abstracts across five research domains, with manual annotations of event segments, triggers, and fine-grained arguments. We define SciIE as a multi-stage EE pipeline: (1) segmenting abstracts into core scientific activities--Background, Method, Result, and Conclusion; and (2) extracting the corresponding triggers and arguments. Experiments with fine-tuned EE models, large language models (LLMs), and human annotators reveal a performance gap, with current models struggling in domains such as sociology and humanities. SciEvent serves as a challenging benchmark and a step toward generalizable, multi-domain SciIE.

Related papers

• WildSci: Advancing Scientific Reasoning from In-the-Wild Literature [50.16160754134139]
  We introduce WildSci, a new dataset of domain-specific science questions automatically synthesized from peer-reviewed literature.<n>By framing complex scientific reasoning tasks in a multiple-choice format, we enable scalable training with well-defined reward signals.<n>Experiments on a suite of scientific benchmarks demonstrate the effectiveness of our dataset and approach.
  arXiv  Detail & Related papers  (2026-01-09T06:35:23Z)
• Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows [203.3527268311731]
  We present an operational SGI definition grounded in the Practical Inquiry Model (PIM)<n>We operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning.<n>Our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.
  arXiv  Detail & Related papers  (2025-12-18T12:44:36Z)
• SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines [112.78540935201558]
  We present a scientific reasoning foundation model that aligns natural language with heterogeneous scientific representations.<n>The model is pretrained on a 206B-token corpus spanning scientific text, pure sequences, and sequence-text pairs, then aligned via SFT on 40M instructions.<n>It supports four capability families, covering up to 103 tasks across: (i) faithful translation between text and scientific formats, (ii) text/knowledge extraction, (iii) property prediction, (iv) property classification, (v) unconditional and conditional sequence generation and design.
  arXiv  Detail & Related papers  (2025-09-25T17:52:06Z)
• SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery [3.779883844533933]
  This paper presents SciGPT, a domain-adapted model for scientific literature understanding and ScienceBench, an open source benchmark tailored to evaluate scientific LLMs.<n>Built on the Qwen3 architecture, SciGPT incorporates three key innovations: (1) low-cost domain distillation via a two-stage pipeline to balance performance and efficiency; (2) a Sparse Mixture-of-Experts attention mechanism that cuts memory consumption by 55% for 32,000 long-token reasoning; and (3) knowledge-aware adaptation integrating domain-specific nuances.<n> Experimental results on ScienceBench show that SciGPT outperforms GPT-4o in core scientific tasks including sequence
  arXiv  Detail & Related papers  (2025-09-09T16:09:19Z)
• A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [221.34650992288505]
  Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
  arXiv  Detail & Related papers  (2025-08-28T18:30:52Z)
• Leveraging Large Language Models for Generating Research Topic Ontologies: A Multi-Disciplinary Study [0.8633013637160062]
  We investigate the capability of several large language models to identify semantic relationships among research topics.<n>Our experiments demonstrate that fine-tuning Mess on PEM-Rel-8K yields excellent performance across all disciplines.
  arXiv  Detail & Related papers  (2025-08-28T11:53:45Z)
• SciTopic: Enhancing Topic Discovery in Scientific Literature through Advanced LLM [19.949137890090814]
  We propose an advanced topic discovery method enhanced by large language models (LLMs) to improve scientific topic identification.<n> Specifically, we build a textual encoder to capture the content from scientific publications, including metadata, title, and abstract.<n>We then construct a space optimization module that integrates entropy-based sampling and triplet tasks guided by LLMs.<n>Experiments conducted on three real-world datasets demonstrate that SciTopic outperforms the state-of-the-art (SOTA) scientific topic discovery methods.
  arXiv  Detail & Related papers  (2025-08-28T07:55:06Z)
• SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents [49.54155332262579]
  We release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations.
  arXiv  Detail & Related papers  (2024-10-28T15:56:49Z)
• EXCEEDS: Extracting Complex Events as Connecting the Dots to Graphs in Scientific Domain [57.56639626657212]
  We construct SciEvents, a large-scale multi-event document-level dataset with a schema tailored for scientific domain. Then, we propose EXCEEDS, a novel end-to-end scientific event extraction framework by storing dense nuggets in a grid matrix. Experimental results demonstrate state-of-the-art performances of EXCEEDS on SciEvents.
  arXiv  Detail & Related papers  (2024-06-20T07:50:37Z)
• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery [68.48094108571432]
  Large language models (LLMs) have revolutionized the way text and other modalities of data are handled. We aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs.
  arXiv  Detail & Related papers  (2024-06-16T08:03:24Z)
• SKT5SciSumm -- Revisiting Extractive-Generative Approach for Multi-Document Scientific Summarization [24.051692189473723]
  We propose SKT5SciSumm - a hybrid framework for multi-document scientific summarization (MDSS) We leverage the Sentence-Transformer version of Scientific Paper Embeddings using Citation-Informed Transformers (SPECTER) to encode and represent textual sentences. We employ the T5 family of models to generate abstractive summaries using extracted sentences.
  arXiv  Detail & Related papers  (2024-02-27T08:33:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.