LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling

• URL: http://arxiv.org/abs/2606.04552v1
• Date: Wed, 03 Jun 2026 07:38:17 GMT
• Title: LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
• Abstract summary: We present LDARNet, a hierarchical genomic foundation model that adapts H-Net-style dynamic chunking from autoregressive generation to masked language modeling.<n>Fine-tuned on 27 tasks from the Nucleotide Transformer and Genomic Benchmarks suites, LDARNet achieves 11/18 wins among compact models and state-of-the-art results on 5 histone modification tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Genomic foundation models increasingly adopt large language model architectures, yet almost universally rely on fixed tokenization schemes such as $k$-mers, BPE, or single nucleotides, which impose arbitrary sequence boundaries that may obscure biologically relevant structure. We present LDARNet, a 120M-parameter hierarchical genomic foundation model that adapts H-Net-style dynamic chunking from autoregressive generation to masked language modeling, combining BiMamba-2 state-space layers with local attention, bidirectional routing, and a ratio-based regularizer to induce adaptive token boundaries without supervision. Fine-tuned on 27 tasks from the Nucleotide Transformer and Genomic Benchmarks suites, LDARNet achieves 11/18 wins among compact models ($<$300M parameters) and state-of-the-art results on 5 histone modification tasks, outperforming models up to 20$\times$ larger. A FLOPs-matched controlled experiment isolates learned routing as the source of these gains: learned boundaries beat fixed-grid boundaries by up to 14 percentage points on histone tasks at identical compute. Nucleotide-resolution analysis further shows that the learned boundaries align with canonical promoter motifs and splice junctions without supervision, providing a biological interpretation for adaptive tokenization in genomic foundation models.

Related papers

• LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence [54.21298601916397]
  LimiX-2 is developed through model and data scaling guided by our previously established scaling laws.<n>LimiX-2 adopts the Contextual Mechanism Networks (CMNs) paradigm and is pretrained with Context-Conditional Masked Modeling ( CCMM)<n> CCMM shifts the organizing principle of in-context learning from target-centric prediction to mechanism-oriented joint modeling.
  arXiv  Detail & Related papers  (2026-09-15T17:30:02Z)
• Reinforcement Learning for Microcanonical Graph Ensemble with Assortativity Constraints [1.0896567381206717]
  How network structure determines function is a fundamental question, and it can be investigated by graph ensembles with precisely controlled structural properties.<n>Here we introduce the Deep Microcanonical Graph Generator (DMGG), a reinforcement learning framework that transforms any given graph through degree-preserving rewirings.
  arXiv  Detail & Related papers  (2026-05-22T06:57:41Z)
• dnaHNet: A Scalable and Hierarchical Foundation Model for Genomic Sequence Learning [16.347455063831067]
  dnaHNet is a tokenizer-free autoregressive model that segments and models genomic sequences end-to-end.<n>Pretrained on prokaryotic genomes, dnaHNet outperforms leading architectures including StripedHyena2 in scaling and efficiency.<n>On zero-shot tasks, dnaHNet achieves superior performance in predicting protein variant fitness and gene essentiality, while automatically discovering hierarchical biological structures without supervision.
  arXiv  Detail & Related papers  (2026-02-11T07:48:21Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• ReDiSC: A Reparameterized Masked Diffusion Model for Scalable Node Classification with Structured Predictions [64.17845687013434]
  We propose ReDiSC, a structured diffusion model for structured node classification.<n>We show that ReDiSC achieves superior or highly competitive performance compared to state-of-the-art GNN, label propagation, and diffusion-based baselines.<n> Notably, ReDiSC scales effectively to large-scale datasets on which previous structured diffusion methods fail due to computational constraints.
  arXiv  Detail & Related papers  (2025-07-19T04:46:53Z)
• A Lightweight Gradient-based Causal Discovery Framework with Applications to Complex Industrial Processes [5.351842005323824]
  We propose Gradient Regularization-based Neural Granger Causality (GRNGC)<n> GRNGC requires only one time series prediction model and applies $L_1$ regularization to the gradient between model's input and output to infer Granger causality.<n> Numerical simulations on DREAM, Lorenz-96, fMRI, and CausalTime show that GRNGC outperforms existing baselines and significantly reduces computational overhead.
  arXiv  Detail & Related papers  (2025-07-15T10:35:29Z)
• JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model [7.8918969994977575]
  Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types.<n>We introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm.<n>JanusDNA processes up to 1 million base pairs at single nucleotide resolution on a single 80GB GPU.
  arXiv  Detail & Related papers  (2025-05-22T20:10:55Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures [179.66117325866585]
  We investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration.
  arXiv  Detail & Related papers  (2020-06-29T17:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.