Molecule Attention Transformer

• URL: http://arxiv.org/abs/2002.08264v1
• Date: Wed, 19 Feb 2020 16:14:48 GMT
• Title: Molecule Attention Transformer
• Authors: {\L}ukasz Maziarka, Tomasz Danel, S{\l}awomir Mucha, Krzysztof Rataj, Jacek Tabor, Stanis{\l}aw Jastrz\k{e}bski
• Abstract summary: We propose Molecule Attention Transformer (MAT) to design a single neural network architecture that performs competitively across a range of molecule property prediction tasks. Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure.
• Score: 5.441166835871135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry. To move towards this goal, we propose Molecule Attention Transformer (MAT). Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure. Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks. Most importantly, with a simple self-supervised pretraining, MAT requires tuning of only a few hyperparameter values to achieve state-of-the-art performance on downstream tasks. Finally, we show that attention weights learned by MAT are interpretable from the chemical point of view.

Related papers

• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Contrastive Dual-Interaction Graph Neural Network for Molecular Property Prediction [0.0]
  We introduce DIG-Mol, a novel self-supervised graph neural network framework for molecular property prediction. DIG-Mol integrates a momentum distillation network with two interconnected networks to efficiently improve molecular characterization. We have established DIG-Mol's state-of-the-art performance through extensive experimental evaluation in a variety of molecular property prediction tasks.
  arXiv  Detail & Related papers  (2024-05-04T10:09:27Z)
• Asymmetric Contrastive Multimodal Learning for Advancing Chemical Understanding [18.217304893881405]
  We introduce Asymmetric Contrastive Multimodal Learning (ACML) as a novel approach tailored for molecules. ACML harnesses the power of effective asymmetric contrastive learning to seamlessly transfer information from various chemical modalities to molecular graph representations. We demonstrate the effectiveness of this framework through large-scale cross-modality retrieval and isomer discrimination tasks.
  arXiv  Detail & Related papers  (2023-11-11T01:58:45Z)
• Implicit Geometry and Interaction Embeddings Improve Few-Shot Molecular Property Prediction [53.06671763877109]
  We develop molecular embeddings that encode complex molecular characteristics to improve the performance of few-shot molecular property prediction. Our approach leverages large amounts of synthetic data, namely the results of molecular docking calculations. On multiple molecular property prediction benchmarks, training from the embedding space substantially improves Multi-Task, MAML, and Prototypical Network few-shot learning performance.
  arXiv  Detail & Related papers  (2023-02-04T01:32:40Z)
• MolCPT: Molecule Continuous Prompt Tuning to Generalize Molecular Representation Learning [77.31492888819935]
  We propose a novel paradigm of "pre-train, prompt, fine-tune" for molecular representation learning, named molecule continuous prompt tuning (MolCPT) MolCPT defines a motif prompting function that uses the pre-trained model to project the standalone input into an expressive prompt. Experiments on several benchmark datasets show that MolCPT efficiently generalizes pre-trained GNNs for molecular property prediction.
  arXiv  Detail & Related papers  (2022-12-20T19:32:30Z)
• BatmanNet: Bi-branch Masked Graph Transformer Autoencoder for Molecular Representation [21.03650456372902]
  We propose a novel bi-branch masked graph transformer autoencoder (BatmanNet) to learn molecular representations. BatmanNet features two tailored complementary and asymmetric graph autoencoders to reconstruct the missing nodes and edges. It achieves state-of-the-art results for multiple drug discovery tasks, including molecular properties prediction, drug-drug interaction, and drug-target interaction.
  arXiv  Detail & Related papers  (2022-11-25T09:44:28Z)
• Relative Molecule Self-Attention Transformer [4.020171169198032]
  Relative Molecule Attention Transformer (R-MAT) is a novel Transformer-based model based on the developed self-attention layer that achieves state-of-the-art or very competitive results across awide range of molecule property prediction tasks. Our main contribution is Relative Molecule Attention Transformer (R-MAT): a novel Transformer-based model based on the developed self-attention layer that achieves state-of-the-art or very competitive results across awide range of molecule property prediction tasks.
  arXiv  Detail & Related papers  (2021-10-12T09:05:26Z)
• Do Large Scale Molecular Language Representations Capture Important Structural Information? [31.76876206167457]
  We present molecular embeddings obtained by training an efficient transformer encoder model, referred to as MoLFormer. Experiments show that the learned molecular representation performs competitively, when compared to graph-based and fingerprint-based supervised learning baselines.
  arXiv  Detail & Related papers  (2021-06-17T14:33:55Z)
• Dual-view Molecule Pre-training [186.07333992384287]
  Dual-view molecule pre-training can effectively combine the strengths of both types of molecule representations. DMP is tested on nine molecular property prediction tasks and achieves state-of-the-art performances on seven of them.
  arXiv  Detail & Related papers  (2021-06-17T03:58:38Z)
• MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization [51.00815310242277]
  generative models and reinforcement learning approaches made initial success, but still face difficulties in simultaneously optimizing multiple drug properties. We propose the MultI-constraint MOlecule SAmpling (MIMOSA) approach, a sampling framework to use input molecule as an initial guess and sample molecules from the target distribution.
  arXiv  Detail & Related papers  (2020-10-05T20:18:42Z)
• Self-Supervised Graph Transformer on Large-Scale Molecular Data [73.3448373618865]
  We propose a novel framework, GROVER, for molecular representation learning. GROVER can learn rich structural and semantic information of molecules from enormous unlabelled molecular data. We pre-train GROVER with 100 million parameters on 10 million unlabelled molecules -- the biggest GNN and the largest training dataset in molecular representation learning.
  arXiv  Detail & Related papers  (2020-06-18T08:37:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.