SAE-RNA: A Sparse Autoencoder Model for Interpreting RNA Language Model Representations

• URL: http://arxiv.org/abs/2510.02734v1
• Date: Fri, 03 Oct 2025 05:34:59 GMT
• Title: SAE-RNA: A Sparse Autoencoder Model for Interpreting RNA Language Model Representations
• Authors: Taehan Kim, Sangdae Nam,
• Abstract summary: We present SAE- RNA, interpretability model that analyzes RiNALMo representations and maps them to known human-level biological features.<n>Our work frames RNA interpretability as concept discovery in pretrained embeddings, without end-to-end retraining.
• Score: 3.2228025627337864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning, particularly with the advancement of Large Language Models, has transformed biomolecular modeling, with protein advances (e.g., ESM) inspiring emerging RNA language models such as RiNALMo. Yet how and what these RNA Language Models internally encode about messenger RNA (mRNA) or non-coding RNA (ncRNA) families remains unclear. We present SAE- RNA, interpretability model that analyzes RiNALMo representations and maps them to known human-level biological features. Our work frames RNA interpretability as concept discovery in pretrained embeddings, without end-to-end retraining, and provides practical tools to probe what RNA LMs may encode about ncRNA families. The model can be extended to close comparisons between RNA groups, and supporting hypothesis generation about previously unrecognized relationships.

Related papers

• TRIDENT: A Trimodal Cascade Generative Framework for Drug and RNA-Conditioned Cellular Morphology Synthesis [56.9460577864211]
  TRIDENT is a cascade generative framework that synthesizes realistic cellular morphology by conditioning on both the perturbation and the corresponding gene expression profile.<n> TRIDENT significantly outperforms state-of-the-art approaches, achieving up to 7-fold improvement with strong generalization to unseen compounds.
  arXiv  Detail & Related papers  (2025-11-23T04:43:27Z)
• RNAMunin: A Deep Machine Learning Model for Non-coding RNA Discovery [0.0]
  Non-coding RNAs (ncRNAs) are critical for regulating bacterial and archaeal physiology, stress response and metabolism.<n>This paper presents RNAMunin, a machine learning (ML) model that is capable of finding ncRNAs using genomic sequence alone.<n> RNAMunin is trained on Rfam sequences extracted from approximately 60 Gbp of long read metagenomes from 16 San Francisco Estuary samples.
  arXiv  Detail & Related papers  (2025-07-16T06:33:50Z)
• A Comparative Review of RNA Language Models [12.899321790353238]
  We divided RNA LMs into three classes (pretrained on multiple RNA types, specific-purpose RNAs, and LMs that unify RNA with DNA or proteins or both)<n>We compared 13 RNA LMs along with 3 DNA and 1 protein LMs as controls in zero-shot prediction of RNA secondary structure and functional classification.<n>Results shows that the models doing well on secondary structure prediction often perform worse in function classification or vice versa, suggesting that more balanced unsupervised training is needed.
  arXiv  Detail & Related papers  (2025-05-14T02:40:13Z)
• Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification [55.98854157265578]
  Life-Code is a comprehensive framework that spans different biological functions.<n>We propose a unified pipeline to integrate multi-omics data by reverse-transcribing RNA and reverse-translating amino acids into nucleotide-based sequences.<n>Life-Code achieves state-of-the-art results on various tasks across three omics, highlighting its potential for advancing multi-omics analysis and interpretation.
  arXiv  Detail & Related papers  (2025-02-11T06:53:59Z)
• RNA-GPT: Multimodal Generative System for RNA Sequence Understanding [6.611255836269348]
  RNAs are essential molecules that carry genetic information vital for life. Despite this importance, RNA research is often hindered by the vast literature available on the topic. We introduce RNA-GPT, a multi-modal RNA chat model designed to simplify RNA discovery.
  arXiv  Detail & Related papers  (2024-10-29T06:19:56Z)
• BEACON: Benchmark for Comprehensive RNA Tasks and Language Models [60.02663015002029]
  We introduce the first comprehensive RNA benchmark BEACON (textbfBEnchmtextbfArk for textbfCOmprehensive RtextbfNA Task and Language Models).<n>First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications.<n>Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models.<n>Third, we investigate the vital RNA language model components
  arXiv  Detail & Related papers  (2024-06-14T19:39:19Z)
• RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks [1.1764999317813143]
  We introduce RiboNucleic Acid Language Model (RiNALMo) to unveil the hidden code of RNA. RiNALMo is the largest RNA language model to date, with 650M parameters pre-trained on 36M non-coding RNA sequences.
  arXiv  Detail & Related papers  (2024-02-29T14:50:58Z)
• Description Generation using Variational Auto-Encoders for precursor microRNA [5.6710852973206105]
  We propose a novel framework, which makes use of generative modeling through Vari Auto-Encoders to uncover latent factors of pre-miRNA. Applying the framework to classification, we obtain a high reconstruction and classification performance, while also developing an accurate description.
  arXiv  Detail & Related papers  (2023-11-29T15:41:45Z)
• scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis in Brain [46.39828178736219]
  We introduce scHyena, a foundation model designed to address these challenges and enhance the accuracy of scRNA-seq analysis in the brain. scHyena is equipped with a linear adaptor layer, the positional encoding via gene-embedding, and a bidirectional Hyena operator. This enables us to process full-length scRNA-seq data without losing any information from the raw data.
  arXiv  Detail & Related papers  (2023-10-04T10:30:08Z)
• RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design [65.41144149958208]
  This study aims to systematically construct a data-driven RNA design pipeline. We crafted a benchmark dataset and designed a comprehensive structural modeling approach to represent the complex RNA tertiary structure. We incorporated extracted secondary structures with base pairs as prior knowledge to facilitate the RNA design process.
  arXiv  Detail & Related papers  (2023-01-25T17:19:49Z)
• Accurate RNA 3D structure prediction using a language model-based deep learning approach [50.193512039121984]
  RhoFold+ is an RNA language model-based deep learning method that accurately predicts 3D structures of single-chain RNAs from sequences.<n>RhoFold+ offers a fully automated end-to-end pipeline for RNA 3D structure prediction.
  arXiv  Detail & Related papers  (2022-07-04T17:15:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.