Relative Molecule Self-Attention Transformer

• URL: http://arxiv.org/abs/2110.05841v1
• Date: Tue, 12 Oct 2021 09:05:26 GMT
• Title: Relative Molecule Self-Attention Transformer
• Authors: {\L}ukasz Maziarka, Dawid Majchrowski, Tomasz Danel, Piotr Gai\'nski, Jacek Tabor, Igor Podolak, Pawe{\l} Morkisz, Stanis{\l}aw Jastrz\k{e}bski
• Abstract summary: Relative Molecule Attention Transformer (R-MAT) is a novel Transformer-based model based on the developed self-attention layer that achieves state-of-the-art or very competitive results across awide range of molecule property prediction tasks. Our main contribution is Relative Molecule Attention Transformer (R-MAT): a novel Transformer-based model based on the developed self-attention layer that achieves state-of-the-art or very competitive results across awide range of molecule property prediction tasks.
• Score: 4.020171169198032
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning holds promise to revolutionize molecule property prediction - a central task to drug discovery and many more industries - by enabling data efficient learning from scarce experimental data. Despite significant progress, non-pretrained methods can be still competitive in certain settings. We reason that architecture might be a key bottleneck. In particular, enriching the backbone architecture with domain-specific inductive biases has been key for the success of self-supervised learning in other domains. In this spirit, we methodologically explore the design space of the self-attention mechanism tailored to molecular data. We identify a novel variant of self-attention adapted to processing molecules, inspired by the relative self-attention layer, which involves fusing embedded graph and distance relationships between atoms. Our main contribution is Relative Molecule Attention Transformer (R-MAT): a novel Transformer-based model based on the developed self-attention layer that achieves state-of-the-art or very competitive results across a~wide range of molecule property prediction tasks.

Related papers

• Data-Efficient Molecular Generation with Hierarchical Textual Inversion [48.816943690420224]
  We introduce Hierarchical textual Inversion for Molecular generation (HI-Mol), a novel data-efficient molecular generation method. HI-Mol is inspired by the importance of hierarchical information, e.g., both coarse- and fine-grained features, in understanding the molecule distribution. Compared to the conventional textual inversion method in the image domain using a single-level token embedding, our multi-level token embeddings allow the model to effectively learn the underlying low-shot molecule distribution.
  arXiv  Detail & Related papers  (2024-05-05T08:35:23Z)
• Zero Shot Molecular Generation via Similarity Kernels [0.6597195879147557]
  We present Similarity-based Molecular Generation (SiMGen), a new method for zero shot molecular generation. SiMGen combines a time-dependent similarity kernel with descriptors from a pretrained machine learning force field to generate molecules. We also release an interactive web tool that allows users to generate structures with SiMGen online.
  arXiv  Detail & Related papers  (2024-02-13T17:53:44Z)
• Implicit Geometry and Interaction Embeddings Improve Few-Shot Molecular Property Prediction [53.06671763877109]
  We develop molecular embeddings that encode complex molecular characteristics to improve the performance of few-shot molecular property prediction. Our approach leverages large amounts of synthetic data, namely the results of molecular docking calculations. On multiple molecular property prediction benchmarks, training from the embedding space substantially improves Multi-Task, MAML, and Prototypical Network few-shot learning performance.
  arXiv  Detail & Related papers  (2023-02-04T01:32:40Z)
• Discovery of structure-property relations for molecules via hypothesis-driven active learning over the chemical space [0.0]
  We introduce a novel approach for the active learning over the chemical spaces based on hypothesis learning. We construct the hypotheses on the possible relationships between structures and functionalities of interest based on a small subset of data. This approach combines the elements from the symbolic regression methods such as SISSO and active learning into a single framework.
  arXiv  Detail & Related papers  (2023-01-06T14:22:43Z)
• A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language [63.60376252491507]
  We propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine.
  arXiv  Detail & Related papers  (2022-09-12T00:56:57Z)
• Transferring Chemical and Energetic Knowledge Between Molecular Systems with Machine Learning [5.27145343046974]
  We propose a novel methodology for transferring knowledge obtained from simple molecular systems to a more complex one. We focus on the classification of high and low free-energy states. Our results show a remarkable AUC of 0.92 for transfer learning from tri-alanine to the deca-alanine system.
  arXiv  Detail & Related papers  (2022-05-06T16:21:00Z)
• Geometric Transformer for End-to-End Molecule Properties Prediction [92.28929858529679]
  We introduce a Transformer-based architecture for molecule property prediction, which is able to capture the geometry of the molecule. We modify the classical positional encoder by an initial encoding of the molecule geometry, as well as a learned gated self-attention mechanism.
  arXiv  Detail & Related papers  (2021-10-26T14:14:40Z)
• Deep Molecular Dreaming: Inverse machine learning for de-novo molecular design and interpretability with surjective representations [1.433758865948252]
  We propose PASITHEA, a gradient-based molecule optimization technique from computer vision. It exploits the use of gradients by directly reversing the learning process of a neural network, which is trained to predict real-valued chemical properties. Although our results are preliminary, we observe a shift in distribution of a chosen property during inverse-training, a clear indication of PASITHEA's viability.
  arXiv  Detail & Related papers  (2020-12-17T16:34:59Z)
• Advanced Graph and Sequence Neural Networks for Molecular Property Prediction and Drug Discovery [53.00288162642151]
  We develop MoleculeKit, a suite of comprehensive machine learning tools spanning different computational models and molecular representations. Built on these representations, MoleculeKit includes both deep learning and traditional machine learning methods for graph and sequence data. Results on both online and offline antibiotics discovery and molecular property prediction tasks show that MoleculeKit achieves consistent improvements over prior methods.
  arXiv  Detail & Related papers  (2020-12-02T02:09:31Z)
• ASGN: An Active Semi-supervised Graph Neural Network for Molecular Property Prediction [61.33144688400446]
  We propose a novel framework called Active Semi-supervised Graph Neural Network (ASGN) by incorporating both labeled and unlabeled molecules. In the teacher model, we propose a novel semi-supervised learning method to learn general representation that jointly exploits information from molecular structure and molecular distribution. At last, we proposed a novel active learning strategy in terms of molecular diversities to select informative data during the whole framework learning.
  arXiv  Detail & Related papers  (2020-07-07T04:22:39Z)
• Molecule Attention Transformer [5.441166835871135]
  We propose Molecule Attention Transformer (MAT) to design a single neural network architecture that performs competitively across a range of molecule property prediction tasks. Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure.
  arXiv  Detail & Related papers  (2020-02-19T16:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.