Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching

• URL: http://arxiv.org/abs/2604.08574v1
• Date: Fri, 27 Mar 2026 14:07:13 GMT
• Title: Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching
• Abstract summary: We present a framework for transferring mRNA representations from a genomic foundation model into a much smaller model specialized for mRNA sequences.<n> Embedding-level distillation worked better than logit based methods, which we found unstable.<n>Our results highlight embedding-based distillation of mRNA sequences as an effective training strategy for biological foundation models.
• Score: 0.6256402324137827
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Genomic Foundation Models have recently achieved remarkable results and in-vivo translation capabilities. However these models quickly grow to over a few Billion of parameters and are expensive to run when compute is limited. To overcome this challenge, we present a distillation framework for transferring mRNA representations from a state of the art genomic foundation model into a much smaller model specialized for mRNA sequences, reducing the size by 200-fold. Embedding-level distillation worked better than logit based methods, which we found unstable. Benchmarking on mRNA-bench demonstrates that the distilled model achieves state-of-the-art performance among models of comparable size and competes with larger architectures for mRNA-related tasks. Our results highlight embedding-based distillation of mRNA sequences as an effective training strategy for biological foundation models. This enables similar efficient and scalable sequence modelling in genomics, particularly when large models are computationally challenging or infeasible.

Related papers

• Tiny Autoregressive Recursive Models [48.902738432071935]
  Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI.<n>We propose the Autoregressive TRM and evaluate it on small autoregressive tasks.<n>Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture.
  arXiv  Detail & Related papers  (2026-03-09T08:22:45Z)
• TRIDENT: A Trimodal Cascade Generative Framework for Drug and RNA-Conditioned Cellular Morphology Synthesis [56.9460577864211]
  TRIDENT is a cascade generative framework that synthesizes realistic cellular morphology by conditioning on both the perturbation and the corresponding gene expression profile.<n> TRIDENT significantly outperforms state-of-the-art approaches, achieving up to 7-fold improvement with strong generalization to unseen compounds.
  arXiv  Detail & Related papers  (2025-11-23T04:43:27Z)
• CodonMoE: DNA Language Models for mRNA Analyses [4.046100165562807]
  Genomic language models (gLMs) face a fundamental efficiency challenge: either maintain separate specialized models for each biological modality (DNA and RNA) or develop large multi-modal architectures.<n>We introduce CodonMoE, a lightweight adapter that transforms DNA language models into effective RNA analyzers without RNA-specific pretraining.<n>Our approach provides a principled path toward unifying genomic language modeling, leveraging more abundant DNA data and reducing computational overhead.
  arXiv  Detail & Related papers  (2025-08-06T01:40:12Z)
• HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling [52.58723853697152]
  We propose a Hybrid Architecture Distillation (HAD) approach for DNA sequence modeling.<n>We employ the NTv2-500M as the teacher model and devise a grouping masking strategy.<n>Compared to models with similar parameters, our model achieved excellent performance.
  arXiv  Detail & Related papers  (2025-05-27T07:57:35Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models [0.0]
  understanding and predicting RNA behavior is a challenge due to the complexity of RNA structures and interactions. Current RNA models have yet to match the performance observed in the protein domain. ChaRNABERT is able to reach state-of-the-art performance on several tasks in established benchmarks.
  arXiv  Detail & Related papers  (2024-11-05T21:56:16Z)
• Comprehensive benchmarking of large language models for RNA secondary structure prediction [0.0]
  RNA-LLM uses large datasets of RNA sequences to learn, in a self-supervised way, how to represent each RNA base with a semantically rich numerical vector.<n>Among them, predicting the secondary structure is a fundamental task for uncovering RNA functional mechanisms.<n>We present a comprehensive experimental analysis of several pre-trained RNA-LLM, comparing them for the RNA secondary structure prediction task in a unified deep learning framework.
  arXiv  Detail & Related papers  (2024-10-21T17:12:06Z)
• BEACON: Benchmark for Comprehensive RNA Tasks and Language Models [60.02663015002029]
  We introduce the first comprehensive RNA benchmark BEACON (textbfBEnchmtextbfArk for textbfCOmprehensive RtextbfNA Task and Language Models).<n>First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications.<n>Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models.<n>Third, we investigate the vital RNA language model components
  arXiv  Detail & Related papers  (2024-06-14T19:39:19Z)
• Scalable Deep Learning for RNA Secondary Structure Prediction [38.46798525594529]
  We present the RNAformer, a lean deep learning model using axial attention and recycling in the latent space. Our approach achieves state-of-the-art performance on the popular TS0 benchmark dataset. We show experimentally that the RNAformer can learn a biophysical model of the RNA folding process.
  arXiv  Detail & Related papers  (2023-07-14T12:54:56Z)
• METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals [151.3601429216877]
  We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. We propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO) The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks.
  arXiv  Detail & Related papers  (2022-04-13T21:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.