Insights Into the Inner Workings of Transformer Models for Protein Function Prediction

• URL: http://arxiv.org/abs/2309.03631v2
• Date: Fri, 9 Feb 2024 09:57:57 GMT
• Title: Insights Into the Inner Workings of Transformer Models for Protein Function Prediction
• Authors: Markus Wenzel, Erik Gr\"uner, Nils Strodthoff
• Abstract summary: We explored how explainable artificial intelligence (XAI) can help to shed light into the inner workings of neural networks for protein function prediction. The approach enabled us to identify amino acids in the sequences that the transformers pay particular attention to, and to show that these relevant sequence parts reflect expectations from biology and chemistry.
• Score: 1.1183543438473609
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Motivation: We explored how explainable artificial intelligence (XAI) can help to shed light into the inner workings of neural networks for protein function prediction, by extending the widely used XAI method of integrated gradients such that latent representations inside of transformer models, which were finetuned to Gene Ontology term and Enzyme Commission number prediction, can be inspected too. Results: The approach enabled us to identify amino acids in the sequences that the transformers pay particular attention to, and to show that these relevant sequence parts reflect expectations from biology and chemistry, both in the embedding layer and inside of the model, where we identified transformer heads with a statistically significant correspondence of attribution maps with ground truth sequence annotations (e.g. transmembrane regions, active sites) across many proteins. Availability and Implementation: Source code can be accessed at https://github.com/markuswenzel/xai-proteins .

Related papers

• Interpreting and Steering Protein Language Models through Sparse Autoencoders [0.9208007322096533]
  This paper explores the application of sparse autoencoders to interpret the internal representations of protein language models. By performing a statistical analysis on each latent component's relevance to distinct protein annotations, we identify potential interpretations linked to various protein characteristics. We then leverage these insights to guide sequence generation, shortlisting the relevant latent components that can steer the model towards desired targets.
  arXiv  Detail & Related papers  (2025-02-13T10:11:36Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters. The model adheres to the central dogma of molecular biology, accurately generating protein-coding sequences. It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of promoter sequences.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Semantically Rich Local Dataset Generation for Explainable AI in Genomics [0.716879432974126]
  Black box deep learning models trained on genomic sequences excel at predicting the outcomes of different gene regulatory mechanisms. We propose using Genetic Programming to generate datasets by evolving perturbations in sequences that contribute to their semantic diversity.
  arXiv  Detail & Related papers  (2024-07-03T10:31:30Z)
• ETDock: A Novel Equivariant Transformer for Protein-Ligand Docking [36.14826783009814]
  Traditional docking methods rely on scoring functions and deep learning to predict the docking between proteins and drugs. In this paper, we propose a transformer neural network for protein-ligand docking pose prediction. The experimental results on real datasets show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-12T06:23:12Z)
• Transformer Neural Networks Attending to Both Sequence and Structure for Protein Prediction Tasks [3.2235261057020606]
  Recent research has shown that the number of known protein sequences supports learning useful, task-agnostic sequence representations via transformers. We propose a transformer neural network that attends to both sequence and tertiary structure.
  arXiv  Detail & Related papers  (2022-06-17T18:40:19Z)
• Learning Geometrically Disentangled Representations of Protein Folding Simulations [72.03095377508856]
  This work focuses on learning a generative neural network on a structural ensemble of a drug-target protein. Model tasks involve characterizing the distinct structural fluctuations of the protein bound to various drug molecules. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex structural variations.
  arXiv  Detail & Related papers  (2022-05-20T19:38:00Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Deep Learning of High-Order Interactions for Protein Interface Prediction [58.164371994210406]
  We propose to formulate the protein interface prediction as a 2D dense prediction problem. We represent proteins as graphs and employ graph neural networks to learn node features. We incorporate high-order pairwise interactions to generate a 3D tensor containing different pairwise interactions.
  arXiv  Detail & Related papers  (2020-07-18T05:39:35Z)
• BERTology Meets Biology: Interpreting Attention in Protein Language Models [124.8966298974842]
  We demonstrate methods for analyzing protein Transformer models through the lens of attention. We show that attention captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure. We also present a three-dimensional visualization of the interaction between attention and protein structure.
  arXiv  Detail & Related papers  (2020-06-26T21:50:17Z)
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers [42.93754828584075]
  We present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR) Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. It provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence.
  arXiv  Detail & Related papers  (2020-06-05T17:09:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.