RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

• URL: http://arxiv.org/abs/2608.22849v2
• Date: Fri, 28 Aug 2026 14:00:01 GMT
• Title: RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling
• Abstract summary: We present RIBOSPAN, a bidirectional RNA foundation model pretrained with context lengths up to 10,240 nt.<n> RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs.<n>Frozen RNA-type evaluations show that RIBOSPAN learns state-of-the-art RNA representations, with a particularly clear advantage on long RNAs.
• Score: 26.58224679878256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. Native 10K pretraining preserves strong reconstruction at 10,240 tokens and, in a controlled long-context benchmark, maintains strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct short-context extrapolation, but induces substantially greater distal representation diffusion. Frozen RNA-type evaluations show that RIBOSPAN learns state-of-the-art RNA representations, with a particularly clear advantage on long RNAs. Across downstream biological benchmarks, RIBOSPAN emerges as the strongest encoder-only RNA foundation model, achieving state-of-the-art performance in both full-transcript biological property prediction and zero-shot mutation-fitness modeling. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning, biological prediction, and full-transcript mRNA design.

Related papers

• RNA-FM: Flow-Matching Generative Model for Genome-wide RNA-Seq Prediction [48.07329582311906]
  We propose RNA-FM, a flow-matching generative framework for genome-wide bulk RNA-seq prediction from whole-slide images.<n> RNA-FM formulates transcriptomic prediction as a continuous-time conditional transport problem.<n>It enables scalable and biologically interpretable genome-wide gene expression imputation.
  arXiv  Detail & Related papers  (2026-05-12T06:48:58Z)
• Protein-Conditioned Multi-Objective Reinforcement Learning for Full-Length mRNA Design [2.9395143551178333]
  We propose ProMORNA, a multi-objective generation framework that produces complete mRNA transcripts directly from a target sequence.<n>Our approach begins by training a BART-style encoder-decoder model on over 6 million natural protein-mRNA pairs.<n>As a case study, we evaluated ProMORNA on the widely used firefly luciferase target, excluding it from both our supervised training data and the prompt pool.
  arXiv  Detail & Related papers  (2026-05-02T16:05:51Z)
• Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction [44.072730684426794]
  Current benchmarks may overestimate generalization across RNA families.<n>We present the Comprehensive Hierarchical.<n>Non-coding RNA Groups of Non-coding RNAs (CHANRG), a benchmark of 170,083 structurally non-redundant RNAs.
  arXiv  Detail & Related papers  (2026-03-20T20:18:08Z)
• RiboSphere: Learning Unified and Efficient Representations of RNA Structures [57.40815107640066]
  RNA backbones are highly flexible, non-canonical interactions are prevalent, and experimentally determined 3D structures are comparatively scarce.<n>We introduce emphRiboSphere, a framework that learns emphdiscrete geometric representations of RNA by combining vector quantization with flow matching.
  arXiv  Detail & Related papers  (2026-03-20T04:36:35Z)
• Helix-mRNA: A Hybrid Foundation Model For Full Sequence mRNA Therapeutics [3.2508287756500165]
  mRNA-based vaccines have become a major focus in the pharmaceutical industry.<n> optimizing mRNA sequences for those properties remains a complex challenge.<n>We present Helix-mRNA, a structured state-space-based and attention hybrid model to address these challenges.
  arXiv  Detail & Related papers  (2025-02-19T14:51:41Z)
• Comprehensive benchmarking of large language models for RNA secondary structure prediction [0.0]
  RNA-LLM uses large datasets of RNA sequences to learn, in a self-supervised way, how to represent each RNA base with a semantically rich numerical vector.<n>Among them, predicting the secondary structure is a fundamental task for uncovering RNA functional mechanisms.<n>We present a comprehensive experimental analysis of several pre-trained RNA-LLM, comparing them for the RNA secondary structure prediction task in a unified deep learning framework.
  arXiv  Detail & Related papers  (2024-10-21T17:12:06Z)
• Latent Diffusion Models for Controllable RNA Sequence Generation [33.38594748558547]
  RNA is a key intermediary between DNA and protein, exhibiting high sequence diversity and complex three-dimensional structures. We develop a latent diffusion model for generating and optimizing discrete RNA sequences of variable lengths. Empirical results confirm that RNAdiffusion generates non-coding RNAs that align with natural distributions across various biological metrics.
  arXiv  Detail & Related papers  (2024-09-15T19:04:50Z)
• BEACON: Benchmark for Comprehensive RNA Tasks and Language Models [60.02663015002029]
  We introduce the first comprehensive RNA benchmark BEACON (textbfBEnchmtextbfArk for textbfCOmprehensive RtextbfNA Task and Language Models).<n>First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications.<n>Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models.<n>Third, we investigate the vital RNA language model components
  arXiv  Detail & Related papers  (2024-06-14T19:39:19Z)
• RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design [65.41144149958208]
  This study aims to systematically construct a data-driven RNA design pipeline. We crafted a benchmark dataset and designed a comprehensive structural modeling approach to represent the complex RNA tertiary structure. We incorporated extracted secondary structures with base pairs as prior knowledge to facilitate the RNA design process.
  arXiv  Detail & Related papers  (2023-01-25T17:19:49Z)
• Accurate RNA 3D structure prediction using a language model-based deep learning approach [50.193512039121984]
  RhoFold+ is an RNA language model-based deep learning method that accurately predicts 3D structures of single-chain RNAs from sequences.<n>RhoFold+ offers a fully automated end-to-end pipeline for RNA 3D structure prediction.
  arXiv  Detail & Related papers  (2022-07-04T17:15:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.