EvoLen: Evolution-Guided Tokenization for DNA Language Model

• URL: http://arxiv.org/abs/2604.08698v1
• Date: Thu, 09 Apr 2026 18:41:28 GMT
• Title: EvoLen: Evolution-Guided Tokenization for DNA Language Model
• Abstract summary: EvoLen is a tokenizer that combines evolutionary stratification with length-aware decoding to better preserve motif-scale functional sequence units.<n>These results demonstrate that tokenization introduces a critical inductive bias and that incorporating evolutionary information yields more biologically meaningful and interpretable sequence representations.
• Score: 37.47818233836275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tokens serve as the basic units of representation in DNA language models (DNALMs), yet their design remains underexplored. Unlike natural language, DNA lacks inherent token boundaries or predefined compositional rules, making tokenization a fundamental modeling decision rather than a naturally specified one. While existing approaches like byte-pair encoding (BPE) excel at capturing token structures that reflect human-generated linguistic regularities, DNA is organized by biological function and evolutionary constraint rather than linguistic convention. We argue that DNA tokenization should prioritize functional sequence patterns like regulatory motifs-short, recurring segments under evolutionary constraint and typically preserved across species. We incorporate evolutionary information directly into the tokenization process through EvoLen, a tokenizer that combines evolutionary stratification with length-aware decoding to better preserve motif-scale functional sequence units. EvoLen uses cross-species evolutionary signals to group DNA sequences, trains separate BPE tokenizers on each group, merges the resulting vocabularies via a rule prioritizing preserved patterns, and applies length-aware decoding with dynamic programming. Through controlled experiments, EvoLen improves the preservation of functional sequence patterns, differentiation across genomic contexts, and alignment with evolutionary constraint, while matching or outperforming standard BPE across diverse DNALM benchmarks. These results demonstrate that tokenization introduces a critical inductive bias and that incorporating evolutionary information yields more biologically meaningful and interpretable sequence representations.

Related papers

• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging [65.07273789940116]
  This paper introduces a hierarchical architecture that jointly optimize a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks.<n> MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation.
  arXiv  Detail & Related papers  (2025-11-17T19:27:41Z)
• BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects [14.172782866715844]
  Large language models (LLMs) trained on text demonstrated remarkable results on natural language processing (NLP) tasks.<n>DNA differs fundamentally from natural language, as it lacks clearly defined words or a consistent grammar.<n>We pre-train foundation models that effectively integrate sequence variations, in particular Single Nucleotide Polymorphisms (SNPs)<n>Our findings indicate that integrating sequence variations into DNALMs helps capture the biological functions as seen in improvements on all fine-tuning tasks.
  arXiv  Detail & Related papers  (2025-06-26T13:56:32Z)
• evoBPE: Evolutionary Protein Sequence Tokenization [3.4196611972116786]
  Current subword tokenization techniques, primarily developed for natural language processing, often fail to represent protein sequences' complex structural and functional properties adequately.<n>This study introduces evoBPE, a novel tokenization approach that integrates evolutionary mutation patterns into sequence segmentation.<n>evoBPE opens new possibilities for machine learning applications in protein function prediction, structural modeling, and evolutionary analysis.
  arXiv  Detail & Related papers  (2025-03-11T19:19:48Z)
• Regulatory DNA sequence Design with Reinforcement Learning [56.20290878358356]
  We propose a generative approach that leverages reinforcement learning to fine-tune a pre-trained autoregressive model.<n>We evaluate our method on promoter design tasks in two yeast media conditions and enhancer design tasks for three human cell types.
  arXiv  Detail & Related papers  (2025-03-11T02:33:33Z)
• Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification [55.98854157265578]
  Life-Code is a comprehensive framework that spans different biological functions.<n>We propose a unified pipeline to integrate multi-omics data by reverse-transcribing RNA and reverse-translating amino acids into nucleotide-based sequences.<n>Life-Code achieves state-of-the-art results on various tasks across three omics, highlighting its potential for advancing multi-omics analysis and interpretation.
  arXiv  Detail & Related papers  (2025-02-11T06:53:59Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA [44.630039477717624]
  MxDNA is a novel framework where the model autonomously learns an effective DNA tokenization strategy through gradient decent.<n>We show that MxDNA learns unique tokenization strategy distinct to those of previous methods and captures genomic functionalities at a token level during self-supervised pretraining.
  arXiv  Detail & Related papers  (2024-12-18T10:55:43Z)
• Multi-modal Transfer Learning between Biological Foundation Models [2.6545450959042234]
  We propose a multi-modal-specific model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality encoders. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods. We open-source our model, paving the way for new multi-modal gene expression approaches.
  arXiv  Detail & Related papers  (2024-06-20T09:44:53Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.