BERTology Meets Biology: Interpreting Attention in Protein Language Models

• URL: http://arxiv.org/abs/2006.15222v3
• Date: Sun, 28 Mar 2021 21:56:26 GMT
• Title: BERTology Meets Biology: Interpreting Attention in Protein Language Models
• Authors: Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, Nazneen Fatema Rajani
• Abstract summary: We demonstrate methods for analyzing protein Transformer models through the lens of attention. We show that attention captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure. We also present a three-dimensional visualization of the interaction between attention and protein structure.
• Score: 124.8966298974842
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. In this work, we demonstrate a set of methods for analyzing protein Transformer models through the lens of attention. We show that attention: (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We find this behavior to be consistent across three Transformer architectures (BERT, ALBERT, XLNet) and two distinct protein datasets. We also present a three-dimensional visualization of the interaction between attention and protein structure. Code for visualization and analysis is available at https://github.com/salesforce/provis.

Related papers

• Geometric Self-Supervised Pretraining on 3D Protein Structures using Subgraphs [26.727436310732692]
  We propose a novel self-supervised method to pretrain 3D graph neural networks on 3D protein structures. We experimentally show that our proposed pertaining strategy leads to significant improvements up to 6%.
  arXiv  Detail & Related papers  (2024-06-20T09:34:31Z)
• Functional Geometry Guided Protein Sequence and Backbone Structure Co-Design [12.585697288315846]
  We propose a model to jointly design Protein sequence and structure based on automatically detected functional sites. NAEPro is powered by an interleaving network of attention and equivariant layers, which can capture global correlation in a whole sequence. Experimental results show that our model consistently achieves the highest amino acid recovery rate, TM-score, and the lowest RMSD among all competitors.
  arXiv  Detail & Related papers  (2023-10-06T16:08:41Z)
• CrysFormer: Protein Structure Prediction via 3d Patterson Maps and Partial Structure Attention [7.716601082662128]
  A protein's three-dimensional structure often poses nontrivial computation costs. We propose the first transformer-based model that directly utilizes protein crystallography and partial structure information. We demonstrate our method, dubbed textttCrysFormer, can achieve accurate predictions, based on a much smaller dataset size and with reduced computation costs.
  arXiv  Detail & Related papers  (2023-10-05T21:10:22Z)
• Joint Design of Protein Sequence and Structure based on Motifs [11.731131799546489]
  We propose GeoPro, a method to design protein backbone structure and sequence jointly. GeoPro is powered by an equivariant encoder for three-dimensional (3D) backbone structure and a protein sequence decoder guided by 3D geometry. Our method discovers novel $beta$-lactamases and myoglobins which are not present in protein data bank (PDB) and UniProt.
  arXiv  Detail & Related papers  (2023-10-04T03:07:03Z)
• A Latent Diffusion Model for Protein Structure Generation [50.74232632854264]
  We propose a latent diffusion model that can reduce the complexity of protein modeling. We show that our method can effectively generate novel protein backbone structures with high designability and efficiency.
  arXiv  Detail & Related papers  (2023-05-06T19:10:19Z)
• EquiPocket: an E(3)-Equivariant Geometric Graph Neural Network for Ligand Binding Site Prediction [49.674494450107005]
  Predicting the binding sites of target proteins plays a fundamental role in drug discovery. Most existing deep-learning methods consider a protein as a 3D image by spatially clustering its atoms into voxels. This work proposes EquiPocket, an E(3)-equivariant Graph Neural Network (GNN) for binding site prediction.
  arXiv  Detail & Related papers  (2023-02-23T17:18:26Z)
• Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks [68.90692290665648]
  We integrate knowledge learned by protein language models into several state-of-the-art geometric networks. Our findings show an overall improvement of 20% over baselines. Strong evidence indicates that the incorporation of protein language models' knowledge enhances geometric networks' capacity by a significant margin.
  arXiv  Detail & Related papers  (2022-12-07T04:04:04Z)
• Learning Geometrically Disentangled Representations of Protein Folding Simulations [72.03095377508856]
  This work focuses on learning a generative neural network on a structural ensemble of a drug-target protein. Model tasks involve characterizing the distinct structural fluctuations of the protein bound to various drug molecules. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex structural variations.
  arXiv  Detail & Related papers  (2022-05-20T19:38:00Z)
• Independent SE(3)-Equivariant Models for End-to-End Rigid Protein Docking [57.2037357017652]
  We tackle rigid body protein-protein docking, i.e., computationally predicting the 3D structure of a protein-protein complex from the individual unbound structures. We design a novel pairwise-independent SE(3)-equivariant graph matching network to predict the rotation and translation to place one of the proteins at the right docked position. Our model, named EquiDock, approximates the binding pockets and predicts the docking poses using keypoint matching and alignment.
  arXiv  Detail & Related papers  (2021-11-15T18:46:37Z)
• G-VAE, a Geometric Convolutional VAE for ProteinStructure Generation [41.66010308405784]
  We introduce a joint geometric-neural networks approach for comparing, deforming and generating 3D protein structures. Our method is able to generate plausible structures, different from the structures in the training data.
  arXiv  Detail & Related papers  (2021-06-22T16:52:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.