Meaning at the Planck scale? Contextualized word embeddings for doing history, philosophy, and sociology of science

• URL: http://arxiv.org/abs/2411.14073v1
• Date: Thu, 21 Nov 2024 12:38:23 GMT
• Title: Meaning at the Planck scale? Contextualized word embeddings for doing history, philosophy, and sociology of science
• Authors: Arno Simons,
• Abstract summary: Using the term "Planck" as a test case, I evaluate five BERT-based models with varying degrees of domain-specific pretraining. Results demonstrate that the domain-adapted models outperform the general-purpose ones in disambiguating the target term. The study underscores the importance of domain-specific pretraining for analyzing scientific language.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores the potential of contextualized word embeddings (CWEs) as a new tool in the history, philosophy, and sociology of science (HPSS) for studying contextual and evolving meanings of scientific concepts. Using the term "Planck" as a test case, I evaluate five BERT-based models with varying degrees of domain-specific pretraining, including my custom model Astro-HEP-BERT, trained on the Astro-HEP Corpus, a dataset containing 21.84 million paragraphs from 600,000 articles in astrophysics and high-energy physics. For this analysis, I compiled two labeled datasets: (1) the Astro-HEP-Planck Corpus, consisting of 2,900 labeled occurrences of "Planck" sampled from 1,500 paragraphs in the Astro-HEP Corpus, and (2) a physics-related Wikipedia dataset comprising 1,186 labeled occurrences of "Planck" across 885 paragraphs. Results demonstrate that the domain-adapted models outperform the general-purpose ones in disambiguating the target term, predicting its known meanings, and generating high-quality sense clusters, as measured by a novel purity indicator I developed. Additionally, this approach reveals semantic shifts in the target term over three decades in the unlabeled Astro-HEP Corpus, highlighting the emergence of the Planck space mission as a dominant sense. The study underscores the importance of domain-specific pretraining for analyzing scientific language and demonstrates the cost-effectiveness of adapting pretrained models for HPSS research. By offering a scalable and transferable method for modeling the meanings of scientific concepts, CWEs open up new avenues for investigating the socio-historical dynamics of scientific discourses.

Related papers

• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters. Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks. It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics [0.0]
  The project demonstrates the effectiveness and feasibility of adapting a bidirectional transformer for applications in the history, philosophy, and sociology of science. The entire training process was conducted using freely available code, pretrained weights, and text inputs, completed on a single MacBook Pro Laptop. Preliminary evaluations indicate that Astro-HEP-BERT's CWEs perform comparably to domain-adapted BERT models trained from scratch on larger datasets.
  arXiv  Detail & Related papers  (2024-11-22T11:59:15Z)
• SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions [52.35520385083425]
  We present SciDMT, an enhanced and expanded corpus for scientific mention detection. The corpus consists of two components: 1) the SciDMT main corpus, which includes 48 thousand scientific articles with over 1.8 million weakly annotated mention annotations in the format of in-text span, and 2) an evaluation set, which comprises 100 scientific articles manually annotated for evaluation purposes.
  arXiv  Detail & Related papers  (2024-06-20T22:03:21Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [80.49349719239584]
  We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following demonstrations for 54 tasks. SciRIFF is the first dataset focused on extracting and synthesizing information from research literature across a wide range of scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
  This work proposes the first dataset for social science academic hypotheses discovery. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
  arXiv  Detail & Related papers  (2023-09-06T05:19:41Z)
• SciTweets -- A Dataset and Annotation Framework for Detecting Scientific Online Discourse [2.3371548697609303]
  Scientific topics, claims and resources are increasingly debated as part of online discourse. This has led to both significant societal impact and increased interest in scientific online discourse from various disciplines. Research across disciplines currently suffers from a lack of robust definitions of the various forms of science-relatedness.
  arXiv  Detail & Related papers  (2022-06-15T08:14:55Z)
• An Informational Space Based Semantic Analysis for Scientific Texts [62.997667081978825]
  This paper introduces computational methods for semantic analysis and the quantifying the meaning of short scientific texts. The representation of scientific-specific meaning is standardised by replacing the situation representations, rather than psychological properties. The research in this paper conducts the base for the geometric representation of the meaning of texts.
  arXiv  Detail & Related papers  (2022-05-31T11:19:32Z)
• Analyzing Scientific Publications using Domain-Specific Word Embedding and Topic Modelling [0.6308539010172307]
  This paper presents a framework for conducting scientific analyses of academic publications. It combines various techniques of Natural Language Processing, such as word embedding and topic modelling. We propose two novel scientific publication embedding, i.e., PUB-G and PUB-W, which are capable of learning semantic meanings of general as well as domain-specific words.
  arXiv  Detail & Related papers  (2021-12-24T04:25:34Z)
• Semantic Analysis for Automated Evaluation of the Potential Impact of Research Articles [62.997667081978825]
  This paper presents a novel method for vector representation of text meaning based on information theory. We show how this informational semantics is used for text classification on the basis of the Leicester Scientific Corpus. We show that an informational approach to representing the meaning of a text has offered a way to effectively predict the scientific impact of research papers.
  arXiv  Detail & Related papers  (2021-04-26T20:37:13Z)
• Automatic coding of students' writing via Contrastive Representation Learning in the Wasserstein space [6.884245063902909]
  This work is a step towards building a statistical machine learning (ML) method for supporting qualitative analyses of students' writing. We show that the ML algorithm approached the inter-rater reliability of human analysis.
  arXiv  Detail & Related papers  (2020-11-26T16:52:48Z)
• Informational Space of Meaning for Scientific Texts [68.8204255655161]
  We introduce the Meaning Space, in which the meaning of a word is represented by a vector of Relative Information Gain (RIG) about the subject categories that the text belongs to. This new approach is applied to construct the Meaning Space based on Leicester Scientific Corpus (LSC) and Leicester Scientific Dictionary-Core (LScDC) The most informative words are presented for 252 categories. The proposed model based on RIG is shown to have ability to stand out topic-specific words in categories.
  arXiv  Detail & Related papers  (2020-04-28T14:26:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.