Transformer Neural Networks Attending to Both Sequence and Structure for Protein Prediction Tasks

• URL: http://arxiv.org/abs/2206.11057v1
• Date: Fri, 17 Jun 2022 18:40:19 GMT
• Title: Transformer Neural Networks Attending to Both Sequence and Structure for Protein Prediction Tasks
• Authors: Anowarul Kabir, Amarda Shehu
• Abstract summary: Recent research has shown that the number of known protein sequences supports learning useful, task-agnostic sequence representations via transformers. We propose a transformer neural network that attends to both sequence and tertiary structure.
• Score: 3.2235261057020606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increasing number of protein sequences decoded from genomes is opening up new avenues of research on linking protein sequence to function with transformer neural networks. Recent research has shown that the number of known protein sequences supports learning useful, task-agnostic sequence representations via transformers. In this paper, we posit that learning joint sequence-structure representations yields better representations for function-related prediction tasks. We propose a transformer neural network that attends to both sequence and tertiary structure. We show that such joint representations are more powerful than sequence-based representations only, and they yield better performance on superfamily membership across various metrics.

Related papers

• FoldToken: Learning Protein Language via Vector Quantization and Beyond [56.19308144551836]
  We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
  arXiv  Detail & Related papers  (2024-02-04T12:18:51Z)
• Insights Into the Inner Workings of Transformer Models for Protein Function Prediction [1.1183543438473609]
  We explored how explainable artificial intelligence (XAI) can help to shed light into the inner workings of neural networks for protein function prediction. The approach enabled us to identify amino acids in the sequences that the transformers pay particular attention to, and to show that these relevant sequence parts reflect expectations from biology and chemistry.
  arXiv  Detail & Related papers  (2023-09-07T10:54:06Z)
• Classification of network topology and dynamics via sequence characterization [0.1611401281366893]
  We investigate whether the reconstruction of the network via the co-occurrence method is useful to recover both the network topology and agent dynamics generating sequences. We found that the characterization of reconstructed networks provides valuable information regarding the process and topology used to create the sequences.
  arXiv  Detail & Related papers  (2022-06-30T11:05:39Z)
• Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction [6.396288020763144]
  Set-to-sequence problems occur in natural language processing, computer vision and structure prediction. Previous attention-based methods require $n$ layers of their set transformations to explicitly represent $n$-th order relations. We propose a novel neural set encoding method called the Set Interdependence Transformer, capable of relating the set's permutation invariant representation to its elements within sets of any cardinality.
  arXiv  Detail & Related papers  (2022-06-08T07:46:49Z)
• Generative power of a protein language model trained on multiple sequence alignments [0.5639904484784126]
  Computational models starting from large ensembles of evolutionarily related protein sequences capture a representation of protein families. Protein language models trained on multiple sequence alignments, such as MSA Transformer, are highly attractive candidates to this end. We propose and test an iterative method that directly uses the masked language modeling objective to generate sequences using MSA Transformer.
  arXiv  Detail & Related papers  (2022-04-14T16:59:05Z)
• Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks [86.10875837475783]
  Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. Existing neural models have been shown to lack this basic ability in learning symbolic structures. We propose two auxiliary sequence prediction tasks that track the progress of function and argument semantics.
  arXiv  Detail & Related papers  (2021-09-30T16:41:19Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Tensor Representations for Action Recognition [54.710267354274194]
  Human actions in sequences are characterized by the complex interplay between spatial features and their temporal dynamics. We propose novel tensor representations for capturing higher-order relationships between visual features for the task of action recognition. We use higher-order tensors and so-called Eigenvalue Power Normalization (NEP) which have been long speculated to perform spectral detection of higher-order occurrences.
  arXiv  Detail & Related papers  (2020-12-28T17:27:18Z)
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers [42.93754828584075]
  We present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR) Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. It provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence.
  arXiv  Detail & Related papers  (2020-06-05T17:09:16Z)
• Addressing Some Limitations of Transformers with Feedback Memory [51.94640029417114]
  Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. We propose the Feedback Transformer architecture that exposes all previous representations to all future representations. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.
  arXiv  Detail & Related papers  (2020-02-21T16:37:57Z)
• Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching [87.97265483696613]
  We propose a new approach to sequence pair matching with Transformer, by learning head-wise matching representations on multiple levels. Experiments show that our proposed approach can achieve new state-of-the-art performance on multiple tasks.
  arXiv  Detail & Related papers  (2020-01-20T20:02:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.