Evaluating Embedding Frameworks for Scientific Domain

• URL: http://arxiv.org/abs/2510.06244v1
• Date: Fri, 03 Oct 2025 12:53:48 GMT
• Title: Evaluating Embedding Frameworks for Scientific Domain
• Authors: Nouman Ahmed, Ronin Wu, Victor Botev,
• Abstract summary: We build an evaluation suite consisting of several downstream tasks and relevant datasets for each task.<n>We use the constructed evaluation suite to test various word representation and tokenization algorithms.
• Score: 0.04588028371034406
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Finding an optimal word representation algorithm is particularly important in terms of domain specific data, as the same word can have different meanings and hence, different representations depending on the domain and context. While Generative AI and transformer architecture does a great job at generating contextualized embeddings for any given work, they are quite time and compute extensive, especially if we were to pre-train such a model from scratch. In this work, we focus on the scientific domain and finding the optimal word representation algorithm along with the tokenization method that could be used to represent words in the scientific domain. The goal of this research is two fold: 1) finding the optimal word representation and tokenization methods that can be used in downstream scientific domain NLP tasks, and 2) building a comprehensive evaluation suite that could be used to evaluate various word representation and tokenization algorithms (even as new ones are introduced) in the scientific domain. To this end, we build an evaluation suite consisting of several downstream tasks and relevant datasets for each task. Furthermore, we use the constructed evaluation suite to test various word representation and tokenization algorithms.

Related papers

• Are we describing the same sound? An analysis of word embedding spaces of expressive piano performance [4.867952721052875]
  We investigate the uncertainty for the domain of characterizations of expressive piano performance. We test five embedding models and their similarity structure for correspondence with the ground truth. The quality of embedding models shows great variability with respect to this task.
  arXiv  Detail & Related papers  (2023-12-31T12:20:03Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Domain-Specific Word Embeddings with Structure Prediction [3.057136788672694]
  We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.
  arXiv  Detail & Related papers  (2022-10-06T12:45:48Z)
• Efficient Hierarchical Domain Adaptation for Pretrained Language Models [77.02962815423658]
  Generative language models are trained on diverse, general domain corpora. We introduce a method to scale domain adaptation to many diverse domains using a computationally efficient adapter approach.
  arXiv  Detail & Related papers  (2021-12-16T11:09:29Z)
• Accelerating Text Mining Using Domain-Specific Stop Word Lists [57.76576681191192]
  We present a novel approach for the automatic extraction of domain-specific words called the hyperplane-based approach. The hyperplane-based approach can significantly reduce text dimensionality by eliminating irrelevant features. Results indicate that the hyperplane-based approach can reduce the dimensionality of the corpus by 90% and outperforms mutual information.
  arXiv  Detail & Related papers  (2020-11-18T17:42:32Z)
• Automated Discovery of Mathematical Definitions in Text with Deep Neural Networks [6.172021438837204]
  This paper focuses on automatic detection of one-sentence definitions in mathematical texts. We apply deep learning methods such as the Convolutional Neural Network (CNN) and the Long Short-Term Memory network (LSTM) We also present a new dataset for definition extraction from mathematical texts.
  arXiv  Detail & Related papers  (2020-11-09T15:57:53Z)
• Meta-Learning for Domain Generalization in Semantic Parsing [124.32975734073949]
  We use a meta-learning framework which targets zero-shot domain for semantic parsing. We apply a model-agnostic training algorithm that simulates zero-shot parsing virtual train and test sets from disjoint domains.
  arXiv  Detail & Related papers  (2020-10-22T19:00:36Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)
• Comparative Analysis of Word Embeddings for Capturing Word Similarities [0.0]
  Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks. Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embeddings. selecting the appropriate word embeddings is a perplexing task since the projected embedding space is not intuitive to humans.
  arXiv  Detail & Related papers  (2020-05-08T01:16:03Z)
• BURT: BERT-inspired Universal Representation from Twin Structure [89.82415322763475]
  BURT (BERT inspired Universal Representation from Twin Structure) is capable of generating universal, fixed-size representations for input sequences of any granularity. Our proposed BURT adopts the Siamese network, learning sentence-level representations from natural language inference dataset and word/phrase-level representations from paraphrasing dataset. We evaluate BURT across different granularities of text similarity tasks, including STS tasks, SemEval2013 Task 5(a) and some commonly used word similarity tasks.
  arXiv  Detail & Related papers  (2020-04-29T04:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.