evoBPE: Evolutionary Protein Sequence Tokenization

• URL: http://arxiv.org/abs/2503.08838v1
• Date: Tue, 11 Mar 2025 19:19:48 GMT
• Title: evoBPE: Evolutionary Protein Sequence Tokenization
• Authors: Burak Suyunu, Özdeniz Dolu, Arzucan Özgür,
• Abstract summary: Current subword tokenization techniques, primarily developed for natural language processing, often fail to represent protein sequences' complex structural and functional properties adequately.<n>This study introduces evoBPE, a novel tokenization approach that integrates evolutionary mutation patterns into sequence segmentation.<n>evoBPE opens new possibilities for machine learning applications in protein function prediction, structural modeling, and evolutionary analysis.
• Score: 3.4196611972116786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in computational biology have drawn compelling parallels between protein sequences and linguistic structures, highlighting the need for sophisticated tokenization methods that capture the intricate evolutionary dynamics of protein sequences. Current subword tokenization techniques, primarily developed for natural language processing, often fail to represent protein sequences' complex structural and functional properties adequately. This study introduces evoBPE, a novel tokenization approach that integrates evolutionary mutation patterns into sequence segmentation, addressing critical limitations in existing methods. By leveraging established substitution matrices, evoBPE transcends traditional frequency-based tokenization strategies. The method generates candidate token pairs through biologically informed mutations, evaluating them based on pairwise alignment scores and frequency thresholds. Extensive experiments on human protein sequences show that evoBPE performs better across multiple dimensions. Domain conservation analysis reveals that evoBPE consistently outperforms standard Byte-Pair Encoding, particularly as vocabulary size increases. Furthermore, embedding similarity analysis using ESM-2 suggests that mutation-based token replacements preserve biological sequence properties more effectively than arbitrary substitutions. The research contributes to protein sequence representation by introducing a mutation-aware tokenization method that better captures evolutionary nuances. By bridging computational linguistics and molecular biology, evoBPE opens new possibilities for machine learning applications in protein function prediction, structural modeling, and evolutionary analysis.

Related papers

• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation [97.99658944212675]
  We introduce a novel pre-training strategy for protein foundation models. It emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability.
  arXiv  Detail & Related papers  (2024-10-31T15:22:03Z)
• Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model [3.4494754789770186]
  Deep learning methods for protein modeling have demonstrated superior results at lower costs compared to traditional approaches. In mutation effect prediction, the key to pre-training deep learning models lies in accurately interpreting the complex relationships among protein sequence, structure, and function. This study introduces a retrieval-enhanced protein language model for comprehensive analysis of native properties from sequence and local structural interactions.
  arXiv  Detail & Related papers  (2024-10-28T15:28:51Z)
• Structure Language Models for Protein Conformation Generation [66.42864253026053]
  Traditional physics-based simulation methods often struggle with sampling equilibrium conformations. Deep generative models have shown promise in generating protein conformations as a more efficient alternative. We introduce Structure Language Modeling as a novel framework for efficient protein conformation generation.
  arXiv  Detail & Related papers  (2024-10-24T03:38:51Z)
• Learning to Predict Mutation Effects of Protein-Protein Interactions by Microenvironment-aware Hierarchical Prompt Learning [78.38442423223832]
  We develop a novel codebook pre-training task, namely masked microenvironment modeling. We demonstrate superior performance and training efficiency over state-of-the-art pre-training-based methods in mutation effect prediction.
  arXiv  Detail & Related papers  (2024-05-16T03:53:21Z)
• Endowing Protein Language Models with Structural Knowledge [5.587293092389789]
  We introduce a novel framework that enhances protein language models by integrating protein structural data. The refined model, termed Protein Structure Transformer (PST), is further pretrained on a small protein structure database. PST consistently outperforms the state-of-the-art foundation model for protein sequences, ESM-2, setting a new benchmark in protein function prediction.
  arXiv  Detail & Related papers  (2024-01-26T12:47:54Z)
• Pairing interacting protein sequences using masked language modeling [0.3222802562733787]
  We develop a method to pair interacting protein sequences using protein language models trained on sequence alignments. We exploit the ability of MSA Transformer to fill in masked amino acids in multiple sequence alignments using the surrounding context. We show that it captures inter-chain coevolution while it was trained on single-chain data, which means that it can be used out-of-distribution.
  arXiv  Detail & Related papers  (2023-08-14T13:42:09Z)
• Multi-level Protein Representation Learning for Blind Mutational Effect Prediction [5.207307163958806]
  This paper introduces a novel pre-training framework that cascades sequential and geometric analyzers for protein structures. It guides mutational directions toward desired traits by simulating natural selection on wild-type proteins. We assess the proposed approach using a public database and two new databases for a variety of variant effect prediction tasks.
  arXiv  Detail & Related papers  (2023-06-08T03:00:50Z)
• Learning Geometrically Disentangled Representations of Protein Folding Simulations [72.03095377508856]
  This work focuses on learning a generative neural network on a structural ensemble of a drug-target protein. Model tasks involve characterizing the distinct structural fluctuations of the protein bound to various drug molecules. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex structural variations.
  arXiv  Detail & Related papers  (2022-05-20T19:38:00Z)
• ODBO: Bayesian Optimization with Search Space Prescreening for Directed Protein Evolution [18.726398852721204]
  We propose an efficient, experimental design-oriented closed-loop optimization framework for protein directed evolution. ODBO employs a combination of novel low-dimensional protein encoding strategy and Bayesian optimization enhanced with search space prescreening via outlier detection. We conduct and report four protein directed evolution experiments that substantiate the capability of the proposed framework for finding variants with properties of interest.
  arXiv  Detail & Related papers  (2022-05-19T13:21:31Z)
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models [53.17320541056843]
  We propose a fully-differentiable approach for protein structure optimization, guided by a data-driven generative network. Our EBM-Fold approach can efficiently produce high-quality decoys, compared against traditional Rosetta-based structure optimization routines.
  arXiv  Detail & Related papers  (2021-05-11T03:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.