Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation

• URL: http://arxiv.org/abs/2601.22757v1
• Date: Fri, 30 Jan 2026 09:32:12 GMT
• Title: Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation
• Authors: Dong Xu, Qihua Pan, Sisi Yuan, Jianqiang Li, Zexuan Zhu, Junkai Ji,
• Abstract summary: We investigate the scaling behavior of molecular language models across both pretraining and downstream tasks.<n>Results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer.<n>We release the largest library of molecular language models to date to facilitate future research and development.
• Score: 18.008217765253274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Molecular generative models, often employing GPT-style language modeling on molecular string representations, have shown promising capabilities when scaled to large datasets and model sizes. However, it remains unclear and subject to debate whether these models adhere to predictable scaling laws under fixed computational budgets, which is a crucial understanding for optimally allocating resources between model size, data volume, and molecular representation. In this study, we systematically investigate the scaling behavior of molecular language models across both pretraining and downstream tasks. We train 300 models and conduct over 10,000 experiments, rigorously controlling compute budgets while independently varying model size, number of training tokens, and molecular representation. Our results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer, reveal the substantial impact of molecular representation on performance, and explain previously observed inconsistencies in scaling behavior for molecular generation. Additionally, we publicly release the largest library of molecular language models to date to facilitate future research and development. Code and models are available at https://github.com/SZU-ADDG/MLM-Scaling.

Related papers

• Foundation Models for Discovery and Exploration in Chemical Space [57.97784111110166]
  MIST is a family of molecular foundation models trained on large unlabeled datasets.<n>We demonstrate the ability of these models to solve real-world problems across chemical space.
  arXiv  Detail & Related papers  (2025-10-20T17:56:01Z)
• NovoMolGen: Rethinking Molecular Language Model Pretraining [14.403924658046806]
  We introduce NovoMolGen, a family of transformer-based foundation models pretrained on 1.5 billion molecules for de-novo molecule generation.<n>Through extensive empirical analyses, we identify a weak correlation between performance metrics measured during pretraining and actual downstream performance.<n>NovoMolGen establishes new state-of-the-art results, substantially outperforming prior Mol-LLMs and specialized generative models in both unconstrained and goal-directed molecular generation tasks.
  arXiv  Detail & Related papers  (2025-08-19T00:04:48Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Uni-Mol2: Exploring Molecular Pretraining Model at Scale [27.172011090947823]
  We present Uni-Mol2, an innovative molecular pretraining model that integrates features at the atomic level, graph level, and geometry structure level. We successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date.
  arXiv  Detail & Related papers  (2024-06-21T08:28:54Z)
• Observational Scaling Laws and the Predictability of Language Model Performance [51.2336010244645]
  We propose an observational approach that bypasses model training and instead builds scaling laws from 100 publically available models. We show that several emergent phenomena follow a smooth, sigmoidal behavior and are predictable from small models. We show how to predict the impact of post-training interventions like Chain-of-Thought and Self-Consistency as language model capabilities continue to improve.
  arXiv  Detail & Related papers  (2024-05-17T17:49:44Z)
• Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting [53.44684898432997]
  Generation of molecules with desired chemical and biological properties is critical for drug discovery. We propose a probabilistic generative model to capture the joint distribution of molecules and their properties. Our method achieves very strong performances on various molecule design tasks.
  arXiv  Detail & Related papers  (2023-06-09T03:04:21Z)
• Training Trajectories of Language Models Across Scales [99.38721327771208]
  Scaling up language models has led to unprecedented performance gains. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors?
  arXiv  Detail & Related papers  (2022-12-19T19:16:29Z)
• Supervised Pretraining for Molecular Force Fields and Properties Prediction [16.86839767858162]
  We propose to pretrain neural networks on a dataset of 86 millions of molecules with atom charges and 3D geometries as inputs and molecular energies as labels. Experiments show that, compared to training from scratch, fine-tuning the pretrained model can significantly improve the performance for seven molecular property prediction tasks and two force field tasks.
  arXiv  Detail & Related papers  (2022-11-23T08:36:50Z)
• Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study [27.56700461408765]
  Key elements underlying molecular property prediction remain largely unexplored. We conduct an extensive evaluation of representative models using various representations on the MoleculeNet datasets. In total, we have trained 62,820 models, including 50,220 models on fixed representations, 4,200 models on SMILES sequences and 8,400 models on molecular graphs.
  arXiv  Detail & Related papers  (2022-09-26T14:07:59Z)
• Keeping it Simple: Language Models can learn Complex Molecular Distributions [0.0]
  We introduce several challenging generative modeling tasks by compiling especially complex distributions of molecules. The results demonstrate that language models are powerful generative models, capable of adeptly learning complex molecular distributions.
  arXiv  Detail & Related papers  (2021-12-06T13:40:58Z)
• Learning Neural Generative Dynamics for Molecular Conformation Generation [89.03173504444415]
  We study how to generate molecule conformations (textiti.e., 3D structures) from a molecular graph. We propose a novel probabilistic framework to generate valid and diverse conformations given a molecular graph.
  arXiv  Detail & Related papers  (2021-02-20T03:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.