What Does a Chemical Language Model Know About Molecules?

• URL: http://arxiv.org/abs/2606.23443v1
• Date: Mon, 22 Jun 2026 14:59:57 GMT
• Title: What Does a Chemical Language Model Know About Molecules?
• Abstract summary: Chemical language models (cLMs) are widely assumed to learn surface-level syntactic patterns rather than learning meaningful molecular semantics.<n>Here, we apply sparse autoencoders (SAEs) to MolFormer, an encoder-only cLM, to mechanistically examine how molecular representations are built across layers.
• Score: 0.3499870393443268
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chemical language models (cLMs) are widely assumed to learn surface-level syntactic patterns rather than learning meaningful molecular semantics. Here, we apply sparse autoencoders (SAEs) to MolFormer, an encoder-only cLM, to mechanistically examine how molecular representations are built across layers. We discover that early layers rely on position-tracking latents to parse molecular grammar, while later layers encode atom-in-substructure and pharmacologically relevant features. Additionally, we show that non-canonical SMILES produce more disruptive representation shifts than invalid SMILES, driven by position-latent disruption propagating across layers. To support further exploration, we develop InterMol, an interactive visualizer for SAE activations on molecular strings and structures.

Related papers

• MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models [21.71979155816693]
  Molecular embedding models can serve as foundational infrastructure for computational chemistry and drug discovery.<n>We ask whether multimodal large language models (MLLMs) can serve as emphgeneral molecular embedding models.<n>We introduce textbfMolEmb, a lightweight framework that adapts MLLMs by aligning molecular profiles with textual descriptions in a shared embedding space.
  arXiv  Detail & Related papers  (2026-08-24T09:21:04Z)
• MoleCode unlocks structural intelligence in large language models [22.076506020790106]
  MoleCode is an LLM-native, training-free, graph-explicit molecular language.<n>It makes molecular topology directly readable, editable and auditable within the language context.<n>It also changes how inference is allocated, replacing long reasoning traces devoted to implicit structural reconstruction.
  arXiv  Detail & Related papers  (2026-05-15T17:44:27Z)
• Beyond Learning on Molecules by Weakly Supervising on Molecules [0.155963096715749]
  Our Adaptive Chemical Embedding Model (ACE-Mol) learns from weak motifs labeled with.<n>natural language descriptors that are cheap to compute trivial to scale.<n>ACE-Mol achieves state-of-the-art performance across molecular property prediction benchmarks with interpretable, chemically meaningful representations.
  arXiv  Detail & Related papers  (2026-02-04T16:03:20Z)
• KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledge [73.51130155601824]
  We introduce KnowMol-100K, a large-scale dataset with 100K fine-grained molecular annotations across multiple levels.<n>We also propose chemically-informative molecular representation, effectively addressing limitations in existing molecular representation strategies.<n>KnowMol achieves superior performance across molecular understanding and generation tasks.
  arXiv  Detail & Related papers  (2025-10-22T11:23:58Z)
• mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Molecules [61.161776104062774]
  We propose a modular Chemical-Language Model that understands both natural language descriptions of functions and molecular blocks.<n>mCLM front-loads synthesizability considerations while improving the predicted functions of molecules in a principled manner.
  arXiv  Detail & Related papers  (2025-05-18T22:52:39Z)
• Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model [52.84455878597969]
  Mol-LLaMA is a large molecular language model that grasps the general knowledge centered on molecules.<n>To improve molecular understanding, we propose a module that integrates complementary information from different molecular encoders.
  arXiv  Detail & Related papers  (2025-02-19T05:49:10Z)
• MolMetaLM: a Physicochemical Knowledge-Guided Molecular Meta Language Model [19.458584012046646]
  We propose a novel physicochemical knowledge-guided molecular meta language framework MolMetaLM. We design a molecule-specialized meta language paradigm, formatted as multiple S,P,O> knowledge triples sharing the same S (i.e., molecule) By introducing different molecular knowledge and noises, the meta language paradigm generates tens of thousands of pretraining tasks.
  arXiv  Detail & Related papers  (2024-11-23T09:27:38Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• FARM: Functional Group-Aware Representations for Small Molecules [55.281754551202326]
  We introduce Functional Group-Aware Representations for Small Molecules (FARM)<n>FARM is a novel model designed to bridge the gap between SMILES, natural language, and molecular graphs.<n>We evaluate FARM on the MoleculeNet dataset, where it achieves state-of-the-art performance on 11 out of 13 tasks.
  arXiv  Detail & Related papers  (2024-10-02T23:04:58Z)
• LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models [55.5427001668863]
  We present a novel latent diffusion model dubbed LDMol for text-conditioned molecule generation.<n> Experiments show that LDMol outperforms the existing autoregressive baselines on the text-to-molecule generation benchmark.<n>We show that LDMol can be applied to downstream tasks such as molecule-to-text retrieval and text-guided molecule editing.
  arXiv  Detail & Related papers  (2024-05-28T04:59:13Z)
• Empirical Evidence for the Fragment level Understanding on Drug Molecular Structure of LLMs [16.508471997999496]
  We investigate whether and how language models understand the chemical spatial structure from 1D sequences. The results indicate that language models can understand chemical structures from the perspective of molecular fragments.
  arXiv  Detail & Related papers  (2024-01-15T12:53:58Z)
• SALSA: Semantically-Aware Latent Space Autoencoder [45.39474752148712]
  We propose a transformer-autoencoder modified with a contrastive task to learn graph-to-graph similarity between molecules. We show that the composed training objective leads to a higher quality latent space that is more 1) structurally-aware, 2) semantically continuous, and 3) property-aware.
  arXiv  Detail & Related papers  (2023-10-04T11:34:46Z)
• Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective [53.300288393173204]
  Large Language Models (LLMs) have shown remarkable performance in various cross-modal tasks. In this work, we propose an In-context Few-Shot Molecule Learning paradigm for molecule-caption translation. We evaluate the effectiveness of MolReGPT on molecule-caption translation, including molecule understanding and text-based molecule generation.
  arXiv  Detail & Related papers  (2023-06-11T08:16:25Z)
• Learning Latent Space Energy-Based Prior Model for Molecule Generation [59.875533935578375]
  We learn latent space energy-based prior model with SMILES representation for molecule modeling. Our method is able to generate molecules with validity and uniqueness competitive with state-of-the-art models.
  arXiv  Detail & Related papers  (2020-10-19T09:34:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.