Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language

• URL: http://arxiv.org/abs/2608.03855v2
• Date: Thu, 06 Aug 2026 12:23:11 GMT
• Title: Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language
• Abstract summary: CheMatE is a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language.<n>We train the model using a novel, large-scale corpus of scientific documents that were constructed and curated from FineWeb and ChemPile.<n>This design exposes the model to SMILES-curated scientific literature, enabling bi-semantic understanding.
• Score: 2.9764168176721992
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.

Related papers

• MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models [21.71979155816693]
  Molecular embedding models can serve as foundational infrastructure for computational chemistry and drug discovery.<n>We ask whether multimodal large language models (MLLMs) can serve as emphgeneral molecular embedding models.<n>We introduce textbfMolEmb, a lightweight framework that adapts MLLMs by aligning molecular profiles with textual descriptions in a shared embedding space.
  arXiv  Detail & Related papers  (2026-08-24T09:21:04Z)
• Speaking the Language of Science: Toward a General-Purpose Generative Foundation Model for the Natural Sciences [89.71422932447423]
  LOGOS is a scientific generative language model that unifies heterogeneous tasks across the natural sciences.<n>By representing spatial contact and constraint patterns as discrete tokens, the model captures complex structural interactions in a purely sequential manner.<n>Across diverse tasks, LOGOS consistently matches or outperforms domain-specific baselines.
  arXiv  Detail & Related papers  (2026-06-15T16:14:53Z)
• Local-Global Multimodal Contrastive Learning for Molecular Property Prediction [0.4349640169711269]
  We propose LGM-CL, a local-global multimodal contrastive learning framework that jointly models molecular graphs and textual representations.<n>Experiments on MoleculeNet benchmarks demonstrate that LGM-CL achieves consistent and competitive performance across both classification and regression tasks.
  arXiv  Detail & Related papers  (2026-01-30T06:06:17Z)
• The Tokenization Bottleneck: How Vocabulary Extension Improves Chemistry Representation Learning in Pretrained Language Models [0.0]
  "tokenization bottleneck" hampered the application of large language models to chemistry.<n>This paper introduces a principled methodology to resolve this bottleneck by unifying the representation of natural language and molecular structures within a single model.
  arXiv  Detail & Related papers  (2025-11-18T11:12:35Z)
• Thinking like a CHEMIST: Combined Heterogeneous Embedding Model Integrating Structure and Tokens [42.203344899915464]
  We present a novel approach that decomposes molecules into substructures and computes descriptor-based representations for these fragments.<n>We use this substructure and descriptor data as input for language model and also propose a bimodal architecture that integrates this language model with graph-based models.
  arXiv  Detail & Related papers  (2025-02-25T08:53:18Z)
• Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction [9.325818199739759]
  We investigate transformer-based language models, including RoBERTa, T5, Llama-3, and MatSciBERT, for predicting band gaps of semiconductor materials.<n>The inputs encode key material features, such as chemical composition, crystal system, space group, and other structural and electronic properties.
  arXiv  Detail & Related papers  (2025-01-07T00:56:26Z)
• Large Concept Models: Language Modeling in a Sentence Representation Space [62.73366944266477]
  We present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a concept.<n> Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow.<n>We show that our model exhibits impressive zero-shot generalization performance to many languages.
  arXiv  Detail & Related papers  (2024-12-11T23:36:20Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• A Large Encoder-Decoder Family of Foundation Models For Chemical Language [1.1073864511426255]
  This paper introduces a large encoder-decoder chemical foundation models pre-trained on a curated dataset of 91 million SMILES samples sourced from PubChem. Our experiments across multiple benchmark datasets validate the capacity of the proposed model in providing state-of-the-art results for different tasks.
  arXiv  Detail & Related papers  (2024-07-24T20:30:39Z)
• In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
  We study ICL through the lens of a new family of model problems we term in context language learning (ICLL) We evaluate a diverse set of neural sequence models on regular ICLL tasks.
  arXiv  Detail & Related papers  (2024-01-23T18:59:21Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)
• Reprogramming Language Models for Molecular Representation Learning [65.00999660425731]
  We propose Representation Reprogramming via Dictionary Learning (R2DL) for adversarially reprogramming pretrained language models for molecular learning tasks. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver. R2DL achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data and outperforms the baseline in a limited training-data setting.
  arXiv  Detail & Related papers  (2020-12-07T05:50:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.