From Tokens to Materials: Leveraging Language Models for Scientific Discovery

• URL: http://arxiv.org/abs/2410.16165v2
• Date: Sun, 03 Nov 2024 06:03:56 GMT
• Title: From Tokens to Materials: Leveraging Language Models for Scientific Discovery
• Authors: Yuwei Wan, Tong Xie, Nan Wu, Wenjie Zhang, Chunyu Kit, Bram Hoex,
• Abstract summary: This study investigates the application of language model embeddings to enhance material property prediction in materials science. We demonstrate that domain-specific models, particularly MatBERT, significantly outperform general-purpose models in extracting implicit knowledge from compound names and material properties.
• Score: 12.211984932142537
• License:
• Abstract: Exploring the predictive capabilities of language models in material science is an ongoing interest. This study investigates the application of language model embeddings to enhance material property prediction in materials science. By evaluating various contextual embedding methods and pre-trained models, including Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT), we demonstrate that domain-specific models, particularly MatBERT significantly outperform general-purpose models in extracting implicit knowledge from compound names and material properties. Our findings reveal that information-dense embeddings from the third layer of MatBERT, combined with a context-averaging approach, offer the most effective method for capturing material-property relationships from the scientific literature. We also identify a crucial "tokenizer effect," highlighting the importance of specialized text processing techniques that preserve complete compound names while maintaining consistent token counts. These insights underscore the value of domain-specific training and tokenization in materials science applications and offer a promising pathway for accelerating the discovery and development of new materials through AI-driven approaches.

Related papers

• MaterioMiner -- An ontology-based text mining dataset for extraction of process-structure-property entities [0.0]
  We present the MaterioMiner dataset and the materials ontology where ontological concepts are associated with textual entities. We explore the consistency between the three raters and perform fine-process-trained models to showcase the feasibility of named-process recognition model training.
  arXiv  Detail & Related papers  (2024-08-05T21:42:59Z)
• FecTek: Enhancing Term Weight in Lexicon-Based Retrieval with Feature Context and Term-level Knowledge [54.61068946420894]
  We introduce an innovative method by introducing FEature Context and TErm-level Knowledge modules. To effectively enrich the feature context representations of term weight, the Feature Context Module (FCM) is introduced. We also develop a term-level knowledge guidance module (TKGM) for effectively utilizing term-level knowledge to intelligently guide the modeling process of term weight.
  arXiv  Detail & Related papers  (2024-04-18T12:58:36Z)
• Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction [6.349503549199403]
  We introduce our model Materials Informatics Transformer (MatInFormer) for material property prediction. Specifically, we introduce a novel approach that involves learning the grammar of crystallography through the tokenization of pertinent space group information.
  arXiv  Detail & Related papers  (2023-08-30T18:34:55Z)
• Lessons in Reproducibility: Insights from NLP Studies in Materials Science [4.205692673448206]
  We aim to comprehend these studies from a perspective, acknowledging their significant influence on the field of materials informatics, rather than critiquing them. Our study indicates that both papers offered thorough, tidy and well-documenteds, and clear guidance for model evaluation. We highlight areas for improvement such as to provide access to training data where copyright restrictions permit, more transparency on model architecture and the training process, and specifications of software dependency versions.
  arXiv  Detail & Related papers  (2023-07-28T18:36:42Z)
• Leveraging Language Representation for Material Recommendation, Ranking, and Exploration [0.0]
  We introduce a material discovery framework that uses natural language embeddings derived from language models as representations of compositional and structural features. By applying the framework to thermoelectrics, we demonstrate diversified recommendations of prototype structures and identify under-studied high-performance material spaces.
  arXiv  Detail & Related papers  (2023-05-01T21:58:29Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction [13.924666106089425]
  MatSciBERT is a language model trained on a large corpus of scientific literature published in the materials domain. We show that MatSciBERT outperforms SciBERT on three downstream tasks, namely, abstract classification, named entity recognition, and relation extraction. We also discuss some of the applications of MatSciBERT in the materials domain for extracting information.
  arXiv  Detail & Related papers  (2021-09-30T17:35:02Z)
• Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation [49.89831914386982]
  We propose a unified pre-trained language model (PLM) for all forms of text, including unstructured text, semi-structured text, and well-structured text. Our approach outperforms the pre-training of plain text using only 1/4 of the data.
  arXiv  Detail & Related papers  (2021-09-02T16:05:24Z)
• CoLAKE: Contextualized Language and Knowledge Embedding [81.90416952762803]
  We propose the Contextualized Language and Knowledge Embedding (CoLAKE) CoLAKE jointly learns contextualized representation for both language and knowledge with the extended objective. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks.
  arXiv  Detail & Related papers  (2020-10-01T11:39:32Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)
• A Dependency Syntactic Knowledge Augmented Interactive Architecture for End-to-End Aspect-based Sentiment Analysis [73.74885246830611]
  We propose a novel dependency syntactic knowledge augmented interactive architecture with multi-task learning for end-to-end ABSA. This model is capable of fully exploiting the syntactic knowledge (dependency relations and types) by leveraging a well-designed Dependency Relation Embedded Graph Convolutional Network (DreGcn) Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-04T14:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.