DNACHUNKER: Learnable Tokenization for DNA Language Models

• URL: http://arxiv.org/abs/2601.03019v1
• Date: Tue, 06 Jan 2026 13:46:42 GMT
• Title: DNACHUNKER: Learnable Tokenization for DNA Language Models
• Authors: Taewon Kim, Jihwan Shin, Hyomin Kim, Youngmok Jung, Jonhoon Lee, Won-Chul Lee, Insu Han, Sungsoo Ahn,
• Abstract summary: We propose DNACHUNKER, which integrates a learnable dynamic DNA tokenization mechanism and is trained as a masked language model.<n>We demonstrate the performance of DNACHUNKER by training it on the human reference genome (HG38) and testing it on the Nucleotide Transformer and Genomic benchmarks.
• Score: 27.919576076056146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: DNA language models have emerged as powerful tools for decoding the complex language of DNA sequences. However, the performance of these models is heavily affected by their tokenization strategy, i.e., a method used to parse DNA sequences into a shorter sequence of chunks. In this work, we propose DNACHUNKER, which integrates a learnable dynamic DNA tokenization mechanism and is trained as a masked language model. Adopting the dynamic chunking procedure proposed by H-Net, our model learns to segment sequences into variable-length chunks. This dynamic chunking offers two key advantages: it's resilient to shifts and mutations in the DNA, and it allocates more detail to important functional areas. We demonstrate the performance of DNACHUNKER by training it on the human reference genome (HG38) and testing it on the Nucleotide Transformer and Genomic benchmarks. Further ablative experiments reveal that DNACHUNKER learns tokenization that grasps biological grammar and uses smaller chunks to preserve detail in important functional elements such as promoters and exons, while using larger chunks for repetitive, redundant regions.

Related papers

• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging [65.07273789940116]
  This paper introduces a hierarchical architecture that jointly optimize a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks.<n> MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation.
  arXiv  Detail & Related papers  (2025-11-17T19:27:41Z)
• Regulatory DNA sequence Design with Reinforcement Learning [56.20290878358356]
  We propose a generative approach that leverages reinforcement learning to fine-tune a pre-trained autoregressive model.<n>We evaluate our method on promoter design tasks in two yeast media conditions and enhancer design tasks for three human cell types.
  arXiv  Detail & Related papers  (2025-03-11T02:33:33Z)
• HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model [70.69095062674944]
  We propose HybriDNA, a decoder-only DNA language model that incorporates a hybrid Transformer-Mamba2 architecture.<n>This hybrid design enables HybriDNA to efficiently process DNA sequences up to 131kb in length with single-nucleotide resolution.<n>HybriDNA achieves state-of-the-art performance across 33 DNA understanding datasets curated from the BEND, GUE, and LRB benchmarks.
  arXiv  Detail & Related papers  (2025-02-15T14:23:43Z)
• Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA [44.630039477717624]
  MxDNA is a novel framework where the model autonomously learns an effective DNA tokenization strategy through gradient decent.<n>We show that MxDNA learns unique tokenization strategy distinct to those of previous methods and captures genomic functionalities at a token level during self-supervised pretraining.
  arXiv  Detail & Related papers  (2024-12-18T10:55:43Z)
• DNAHLM -- DNA sequence and Human Language mixed large language Model [0.0]
  This paper introduces a pre-trained model trained on the GPT-2 network, combining DNA sequences and English text. We then convert classification and other downstream tasks into Alpaca format instruction data, and perform instruction fine-tuning. The model has demonstrated its effectiveness in DNA related zero-shot prediction and multitask application.
  arXiv  Detail & Related papers  (2024-10-22T11:51:09Z)
• A Benchmark Dataset for Multimodal Prediction of Enzymatic Function Coupling DNA Sequences and Natural Language [3.384797724820242]
  Predicting gene function from its DNA sequence is a fundamental challenge in biology. Deep learning models have been proposed to embed DNA sequences and predict their enzymatic function. Much of the scientific community's knowledge of biological function is not represented in categorical labels.
  arXiv  Detail & Related papers  (2024-07-21T19:27:43Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• BEND: Benchmarking DNA Language Models on biologically meaningful tasks [7.005668635562045]
  We introduce BEND, a Benchmark for DNA language models, featuring a collection of realistic and biologically meaningful downstream tasks. We find that embeddings from current DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features.
  arXiv  Detail & Related papers  (2023-11-21T12:34:00Z)
• Generative Language Models on Nucleotide Sequences of Human Genes [0.0]
  This study focuses on developing an autoregressive generative language model like GPT-3 for DNA sequences. Because working with whole DNA sequences is challenging without substantial computational resources, we decided to carry out our study on a smaller scale. First of all, we systematically examined an almost entirely unexplored problem and observed that RNNs performed the best. How essential using real-life tasks beyond the classical metrics such as perplexity is observed.
  arXiv  Detail & Related papers  (2023-07-20T06:59:02Z)
• HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution [76.97231739317259]
  We present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data.
  arXiv  Detail & Related papers  (2023-06-27T20:46:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.