3D-Transformer: Molecular Representation with Transformer in 3D Space

• URL: http://arxiv.org/abs/2110.01191v1
• Date: Mon, 4 Oct 2021 05:11:23 GMT
• Title: 3D-Transformer: Molecular Representation with Transformer in 3D Space
• Authors: Fang Wu, Qiang Zhang, Dragomir Radev, Jiyu Cui, Wen Zhang, Huabin Xing, Ningyu Zhang, Huajun Chen
• Abstract summary: 3D-Transformer is a variant of the Transformer for molecular representations that incorporates 3D spatial information. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task.
• Score: 11.947499562836953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science, and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

Related papers

• 3D-Mol: A Novel Contrastive Learning Framework for Molecular Property Prediction with 3D Information [1.1777304970289215]
  3D-Mol is a novel approach designed for more accurate spatial structure representation. It deconstructs molecules into three hierarchical graphs to better extract geometric information. We compare 3D-Mol with various state-of-the-art baselines on 7 benchmarks and demonstrate our outstanding performance.
  arXiv  Detail & Related papers  (2023-09-28T10:05:37Z)
• MUDiff: Unified Diffusion for Complete Molecule Generation [104.7021929437504]
  We present a new model for generating a comprehensive representation of molecules, including atom features, 2D discrete molecule structures, and 3D continuous molecule coordinates. We propose a novel graph transformer architecture to denoise the diffusion process. Our model is a promising approach for designing stable and diverse molecules and can be applied to a wide range of tasks in molecular modeling.
  arXiv  Detail & Related papers  (2023-04-28T04:25:57Z)
• Geometry-Complete Diffusion for 3D Molecule Generation and Optimization [3.8366697175402225]
  We introduce the Geometry-Complete Diffusion Model (GCDM) for 3D molecule generation. GCDM outperforms existing 3D molecular diffusion models by significant margins across conditional and unconditional settings. We also show that GCDM's geometric features can be repurposed to consistently optimize the geometry and chemical composition of existing 3D molecules.
  arXiv  Detail & Related papers  (2023-02-08T20:01:51Z)
• Molecular Geometry-aware Transformer for accurate 3D Atomic System modeling [51.83761266429285]
  We propose a novel Transformer architecture that takes nodes (atoms) and edges (bonds and nonbonding atom pairs) as inputs and models the interactions among them. Moleformer achieves state-of-the-art on the initial state to relaxed energy prediction of OC20 and is very competitive in QM9 on predicting quantum chemical properties.
  arXiv  Detail & Related papers  (2023-02-02T03:49:57Z)
• One Transformer Can Understand Both 2D & 3D Molecular Data [94.93514673086631]
  We develop a novel Transformer-based Molecular model called Transformer-M. It can take molecular data of 2D or 3D formats as input and generate meaningful semantic representations. All empirical results show that Transformer-M can simultaneously achieve strong performance on 2D and 3D tasks.
  arXiv  Detail & Related papers  (2022-10-04T17:30:31Z)
• Geometric Transformer for End-to-End Molecule Properties Prediction [92.28929858529679]
  We introduce a Transformer-based architecture for molecule property prediction, which is able to capture the geometry of the molecule. We modify the classical positional encoder by an initial encoding of the molecule geometry, as well as a learned gated self-attention mechanism.
  arXiv  Detail & Related papers  (2021-10-26T14:14:40Z)
• 3D Infomax improves GNNs for Molecular Property Prediction [1.9703625025720701]
  We propose pre-training a model to reason about the geometry of molecules given only their 2D molecular graphs. We show that 3D pre-training provides significant improvements for a wide range of properties.
  arXiv  Detail & Related papers  (2021-10-08T13:30:49Z)
• Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs [79.06686274377009]
  We develop a benchmark, known as Molecule3D, that includes a dataset with precise ground-state geometries of approximately 4 million molecules. We implement two baseline methods that either predict the pairwise distance between atoms or atom coordinates in 3D space. Our method can achieve comparable prediction accuracy but with much smaller computational costs.
  arXiv  Detail & Related papers  (2021-09-30T22:09:28Z)
• ATOM3D: Tasks On Molecules in Three Dimensions [91.72138447636769]
  Deep neural networks have recently gained significant attention. In this work we present ATOM3D, a collection of both novel and existing datasets spanning several key classes of biomolecules. We develop three-dimensional molecular learning networks for each of these tasks, finding that they consistently improve performance.
  arXiv  Detail & Related papers  (2020-12-07T20:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.