JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model

• URL: http://arxiv.org/abs/2505.17257v2
• Date: Mon, 02 Jun 2025 22:18:29 GMT
• Title: JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
• Authors: Qihao Duan, Bingding Huang, Zhenqiao Song, Irina Lehmann, Lei Gu, Roland Eils, Benjamin Wild,
• Abstract summary: Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types.<n>We introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm.<n>JanusDNA processes up to 1 million base pairs at single nucleotide resolution on a single 80GB GPU.
• Score: 1.6128508494592848
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic sequences. However, adapting LLMs to genomics presents significant challenges. Capturing complex genomic interactions requires modeling long-range dependencies within DNA sequences, where interactions often span over 10,000 base pairs, even within a single gene, posing substantial computational burdens under conventional model architectures and training paradigms. Moreover, standard LLM training approaches are suboptimal for DNA: autoregressive training, while efficient, supports only unidirectional understanding. However, DNA is inherently bidirectional, e.g., bidirectional promoters regulate transcription in both directions and account for nearly 11% of human gene expression. Masked language models (MLMs) allow bidirectional understanding but are inefficient, as only masked tokens contribute to the loss per step. To address these limitations, we introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm that combines the optimization efficiency of autoregressive modeling with the bidirectional comprehension of masked modeling. JanusDNA adopts a hybrid Mamba, Attention and Mixture of Experts (MoE) architecture, combining long-range modeling of Attention with efficient sequential learning of Mamba. MoE layers further scale model capacity via sparse activation while keeping computational cost low. Notably, JanusDNA processes up to 1 million base pairs at single nucleotide resolution on a single 80GB GPU. Extensive experiments and ablations show JanusDNA achieves new SOTA results on three genomic representation benchmarks, outperforming models with 250x more activated parameters. Code: https://github.com/Qihao-Duan/JanusDNA

Related papers

• CodonMoE: DNA Language Models for mRNA Analyses [4.046100165562807]
  Genomic language models (gLMs) face a fundamental efficiency challenge: either maintain separate specialized models for each biological modality (DNA and RNA) or develop large multi-modal architectures.<n>We introduce CodonMoE, a lightweight adapter that transforms DNA language models into effective RNA analyzers without RNA-specific pretraining.<n>Our approach provides a principled path toward unifying genomic language modeling, leveraging more abundant DNA data and reducing computational overhead.
  arXiv  Detail & Related papers  (2025-08-06T01:40:12Z)
• Bidirectional Mamba for Single-Cell Data: Efficient Context Learning with Biological Fidelity [0.39945675027960637]
  We introduce GeneMamba, a scalable and efficient foundation model for single-cell transcriptomics built on state space modeling.<n>GeneMamba captures bidirectional gene context with linear-time complexity, offering substantial computational gains over transformer baselines.<n>We evaluate GeneMamba across diverse tasks, including multi-batch integration, cell type annotation, and gene-gene correlation, demonstrating strong performance, interpretability, and robustness.
  arXiv  Detail & Related papers  (2025-04-22T20:34:47Z)
• Regulatory DNA sequence Design with Reinforcement Learning [56.20290878358356]
  We propose a generative approach that leverages reinforcement learning to fine-tune a pre-trained autoregressive model.<n>We evaluate our method on promoter design tasks in two yeast media conditions and enhancer design tasks for three human cell types.
  arXiv  Detail & Related papers  (2025-03-11T02:33:33Z)
• Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale [68.6602625868888]
  We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations.<n>Operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression.<n>We train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids.
  arXiv  Detail & Related papers  (2025-02-25T19:47:20Z)
• HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model [70.69095062674944]
  We propose HybriDNA, a decoder-only DNA language model that incorporates a hybrid Transformer-Mamba2 architecture.<n>This hybrid design enables HybriDNA to efficiently process DNA sequences up to 131kb in length with single-nucleotide resolution.<n>HybriDNA achieves state-of-the-art performance across 33 DNA understanding datasets curated from the BEND, GUE, and LRB benchmarks.
  arXiv  Detail & Related papers  (2025-02-15T14:23:43Z)
• Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA [44.630039477717624]
  MxDNA is a novel framework where the model autonomously learns an effective DNA tokenization strategy through gradient decent.<n>We show that MxDNA learns unique tokenization strategy distinct to those of previous methods and captures genomic functionalities at a token level during self-supervised pretraining.
  arXiv  Detail & Related papers  (2024-12-18T10:55:43Z)
• DNAHLM -- DNA sequence and Human Language mixed large language Model [0.0]
  This paper introduces a pre-trained model trained on the GPT-2 network, combining DNA sequences and English text. We then convert classification and other downstream tasks into Alpaca format instruction data, and perform instruction fine-tuning. The model has demonstrated its effectiveness in DNA related zero-shot prediction and multitask application.
  arXiv  Detail & Related papers  (2024-10-22T11:51:09Z)
• Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling [36.37643634126816]
  Long-range token interactions, the effects of upstream and downstream regions of the genome, and the reverse complementarity of DNA are studied. Here, we propose an architecture motivated by these challenges that builds off the long-range Mamba block. We use MambaDNA as the basis of Caduceus, the first family of RC equivariant bi-directional long-range DNA language models.
  arXiv  Detail & Related papers  (2024-03-05T01:42:51Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution [76.97231739317259]
  We present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data.
  arXiv  Detail & Related papers  (2023-06-27T20:46:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.