Embedding Models for Supervised Automatic Extraction and Classification
of Named Entities in Scientific Acknowledgements
- URL: http://arxiv.org/abs/2307.13377v1
- Date: Tue, 25 Jul 2023 09:51:17 GMT
- Title: Embedding Models for Supervised Automatic Extraction and Classification
of Named Entities in Scientific Acknowledgements
- Authors: Nina Smirnova and Philipp Mayr
- Abstract summary: The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities.
The training was conducted using three default Flair NER models with four differently-sized corpora and different versions of the Flair NLP framework.
The model is able to recognize six entity types: funding agency, grant number, individuals, university, corporation, and miscellaneous.
- Score: 5.330844352905488
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Acknowledgments in scientific papers may give an insight into aspects of the
scientific community, such as reward systems, collaboration patterns, and
hidden research trends. The aim of the paper is to evaluate the performance of
different embedding models for the task of automatic extraction and
classification of acknowledged entities from the acknowledgment text in
scientific papers. We trained and implemented a named entity recognition (NER)
task using the Flair NLP framework. The training was conducted using three
default Flair NER models with four differently-sized corpora and different
versions of the Flair NLP framework. The Flair Embeddings model trained on the
medium corpus with the latest FLAIR version showed the best accuracy of 0.79.
Expanding the size of a training corpus from very small to medium size
massively increased the accuracy of all training algorithms, but further
expansion of the training corpus did not bring further improvement. Moreover,
the performance of the model slightly deteriorated. Our model is able to
recognize six entity types: funding agency, grant number, individuals,
university, corporation, and miscellaneous. The model works more precisely for
some entity types than for others; thus, individuals and grant numbers showed a
very good F1-Score over 0.9. Most of the previous works on acknowledgment
analysis were limited by the manual evaluation of data and therefore by the
amount of processed data. This model can be applied for the comprehensive
analysis of acknowledgment texts and may potentially make a great contribution
to the field of automated acknowledgment analysis.
Related papers
- Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored.
We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches.
We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
arXiv Detail & Related papers (2024-10-02T20:48:28Z) - Exploring the Effectiveness of Methods for Persona Extraction [0.0]
The paper presents a study of methods for extracting information about dialogue participants and evaluating their performance in Russian.
To train models for this task, the Multi-Session Chat dataset was translated into Russian using multiple translation models.
A metric based on the F-score concept is presented to evaluate the effectiveness of the extraction models.
arXiv Detail & Related papers (2024-07-12T11:30:10Z) - A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check.
Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models.
The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
arXiv Detail & Related papers (2023-07-25T17:02:38Z) - How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
Resources [117.6496550359768]
This work explores recent advances in instruction-tuning language models on a range of open instruction-following datasets.
We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets.
We evaluate them on their factual knowledge, reasoning, multilinguality, coding, and open-ended instruction following abilities.
arXiv Detail & Related papers (2023-06-07T19:59:23Z) - Preserving Knowledge Invariance: Rethinking Robustness Evaluation of
Open Information Extraction [50.62245481416744]
We present the first benchmark that simulates the evaluation of open information extraction models in the real world.
We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.
By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
arXiv Detail & Related papers (2023-05-23T12:05:09Z) - Evaluation of Embedding Models for Automatic Extraction and
Classification of Acknowledged Entities in Scientific Documents [5.330844352905488]
The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities.
The training was conducted using three default Flair NER models with two differently-sized corpora.
Our model is able to recognize six entity types: funding agency, grant number, individuals, university, corporation and miscellaneous.
arXiv Detail & Related papers (2022-06-22T09:32:28Z) - Automated and Explainable Ontology Extension Based on Deep Learning: A
Case Study in the Chemical Domain [0.9449650062296822]
We present a new methodology for automatic ontology extension for large domains.
We trained a Transformer-based deep learning model on the leaf node from the ChEBI ontology and the classes to which they belong.
The proposed model achieved an overall F1 score of 0.80, an improvement of 6 percentage points over our previous results.
arXiv Detail & Related papers (2021-09-19T19:37:08Z) - Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
We investigate three schemes to improve the model generalization ability for few-shot settings.
We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data.
We create new state-of-the-art results on both few-shot and training-free settings.
arXiv Detail & Related papers (2020-12-29T23:43:16Z) - Rethinking Generalization of Neural Models: A Named Entity Recognition
Case Study [81.11161697133095]
We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives.
Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models.
As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
arXiv Detail & Related papers (2020-01-12T04:33:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.