A standard transformer and attention with linear biases for molecular conformer generation

• URL: http://arxiv.org/abs/2506.19834v1
• Date: Tue, 24 Jun 2025 17:50:49 GMT
• Title: A standard transformer and attention with linear biases for molecular conformer generation
• Authors: Viatcheslav Gurev, Timothy Rumbell,
• Abstract summary: equivariant networks have been designed to generate molecular conformations from 2D molecular graphs.<n>Non-equivariant transformer models have emerged as a viable alternative due to their capability to scale to improve generalization.<n>In this paper, we demonstrate that a well-chosen positional encoding effectively addresses these size limitations.
• Score: 0.16741831494720966
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Sampling low-energy molecular conformations, spatial arrangements of atoms in a molecule, is a critical task for many different calculations performed in the drug discovery and optimization process. Numerous specialized equivariant networks have been designed to generate molecular conformations from 2D molecular graphs. Recently, non-equivariant transformer models have emerged as a viable alternative due to their capability to scale to improve generalization. However, the concern has been that non-equivariant models require a large model size to compensate the lack of equivariant bias. In this paper, we demonstrate that a well-chosen positional encoding effectively addresses these size limitations. A standard transformer model incorporating relative positional encoding for molecular graphs when scaled to 25 million parameters surpasses the current state-of-the-art non-equivariant base model with 64 million parameters on the GEOM-DRUGS benchmark. We implemented relative positional encoding as a negative attention bias that linearly increases with the shortest path distances between graph nodes at varying slopes for different attention heads, similar to ALiBi, a widely adopted relative positional encoding technique in the NLP domain. This architecture has the potential to serve as a foundation for a novel class of generative models for molecular conformations.

Related papers

• Torsional-GFN: a conditional conformation generator for small molecules [75.91029322687771]
  We introduce a conditional GFlowNet specifically designed to sample conformations of molecules proportionally to their Boltzmann distribution.<n>Our work presents a promising avenue for scaling the proposed approach to larger molecular systems.
  arXiv  Detail & Related papers  (2025-07-15T21:53:25Z)
• Sampling 3D Molecular Conformers with Diffusion Transformers [13.536503487456622]
  Diffusion Transformers (DiTs) have demonstrated strong performance in generative modeling.<n>Applying DiTs to molecules introduces novel challenges, such as integrating discrete molecular graph information with continuous 3D geometry.<n>We propose DiTMC, a framework that adapts DiTs to address these challenges through a modular architecture.
  arXiv  Detail & Related papers  (2025-06-18T11:47:59Z)
• E(3)-equivariant models cannot learn chirality: Field-based molecular generation [51.327048911864885]
  Chirality plays a key role in determining drug safety and potency.<n>We introduce a novel field-based representation, proposing reference rotations that replace rotational symmetry constraints.<n>The proposed model captures all molecular geometries including chirality, while still achieving highly competitive performance with E(3)-based methods across standard benchmarking metrics.
  arXiv  Detail & Related papers  (2024-02-24T17:13:58Z)
• Molecular Geometry-aware Transformer for accurate 3D Atomic System modeling [51.83761266429285]
  We propose a novel Transformer architecture that takes nodes (atoms) and edges (bonds and nonbonding atom pairs) as inputs and models the interactions among them. Moleformer achieves state-of-the-art on the initial state to relaxed energy prediction of OC20 and is very competitive in QM9 on predicting quantum chemical properties.
  arXiv  Detail & Related papers  (2023-02-02T03:49:57Z)
• Modular Flows: Differential Molecular Generation [18.41106104201439]
  Flows can generate molecules effectively by inverting the encoding process. Existing flow models require artifactual dequantization or specific node/edge orderings. We develop continuous normalizing E(3)-equivariant flows, based on a system of node ODEs and a graph PDE. Our models can be cast as message-passing temporal networks, and result in superlative performance on the tasks of density estimation and molecular generation.
  arXiv  Detail & Related papers  (2022-10-12T09:08:35Z)
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation [102.85440102147267]
  We propose a novel generative model named GeoDiff for molecular conformation prediction. We show that GeoDiff is superior or comparable to existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-06T09:47:01Z)
• Geometric Transformer for End-to-End Molecule Properties Prediction [92.28929858529679]
  We introduce a Transformer-based architecture for molecule property prediction, which is able to capture the geometry of the molecule. We modify the classical positional encoder by an initial encoding of the molecule geometry, as well as a learned gated self-attention mechanism.
  arXiv  Detail & Related papers  (2021-10-26T14:14:40Z)
• GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles [60.12186997181117]
  Prediction of a molecule's 3D conformer ensemble from the molecular graph holds a key role in areas of cheminformatics and drug discovery. Existing generative models have several drawbacks including lack of modeling important molecular geometry elements. We propose GeoMol, an end-to-end, non-autoregressive and SE(3)-invariant machine learning approach to generate 3D conformers.
  arXiv  Detail & Related papers  (2021-06-08T14:17:59Z)
• Physics-Constrained Predictive Molecular Latent Space Discovery with Graph Scattering Variational Autoencoder [0.0]
  We develop a molecular generative model based on variational inference and graph theory in the small data regime. The model's performance is evaluated by generating molecules with desired target properties.
  arXiv  Detail & Related papers  (2020-09-29T09:05:27Z)
• Conditional Constrained Graph Variational Autoencoders for Molecule Design [70.59828655929194]
  We present Conditional Constrained Graph Variational Autoencoder (CCGVAE), a model that implements this key-idea in a state-of-the-art model. We show improved results on several evaluation metrics on two commonly adopted datasets for molecule generation.
  arXiv  Detail & Related papers  (2020-09-01T21:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.