Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models

• URL: http://arxiv.org/abs/2411.11808v1
• Date: Tue, 05 Nov 2024 21:56:16 GMT
• Title: Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models
• Authors: Adrián Morales-Pastor, Raquel Vázquez-Reza, Miłosz Wieczór, Clàudia Valverde, Manel Gil-Sorribes, Bertran Miquel-Oliver, Álvaro Ciudad, Alexis Molina,
• Abstract summary: understanding and predicting RNA behavior is a challenge due to the complexity of RNA structures and interactions. Current RNA models have yet to match the performance observed in the protein domain. ChaRNABERT is able to reach state-of-the-art performance on several tasks in established benchmarks.
• Score: 0.0
• License:
• Abstract: RNA is a vital biomolecule with numerous roles and functions within cells, and interest in targeting it for therapeutic purposes has grown significantly in recent years. However, fully understanding and predicting RNA behavior, particularly for applications in drug discovery, remains a challenge due to the complexity of RNA structures and interactions. While foundational models in biology have demonstrated success in modeling several biomolecules, especially proteins, achieving similar breakthroughs for RNA has proven more difficult. Current RNA models have yet to match the performance observed in the protein domain, leaving an important gap in computational biology. In this work, we present ChaRNABERT, a suite of sample and parameter-efficient RNA foundational models, that through a learnable tokenization process, are able to reach state-of-the-art performance on several tasks in established benchmarks. We extend its testing in relevant downstream tasks such as RNA-protein and aptamer-protein interaction prediction. Weights and inference code for ChaRNABERT-8M will be provided for academic research use. The other models will be available upon request.

Related papers

• LoRA-BERT: a Natural Language Processing Model for Robust and Accurate Prediction of long non-coding RNAs [11.346750562942345]
  Long non-coding RNAs (lncRNAs) serve as crucial regulators in numerous biological processes. Deep learning-based approaches have been introduced to classify lncRNAs. LoRA-BERT is designed to capture the importance of nucleotide-level information during sequence classification.
  arXiv  Detail & Related papers  (2024-11-11T22:17:01Z)
• Comprehensive benchmarking of large language models for RNA secondary structure prediction [0.0]
  RNA-LLM uses large datasets of RNA sequences to learn, in a self-supervised way, how to represent each RNA base with a semantically rich numerical vector. Among them, predicting the secondary structure is a fundamental task for uncovering RNA functional mechanisms. We present a comprehensive experimental analysis of several pre-trained RNA-LLM, comparing them for the RNA secondary structure prediction task in a unified deep learning framework.
  arXiv  Detail & Related papers  (2024-10-21T17:12:06Z)
• RNACG: A Universal RNA Sequence Conditional Generation model based on Flow-Matching [0.0]
  We develop a universal RNA sequence generation model based on flow matching, namely RNACG. RNACG can accommodate various conditional inputs and is portable, enabling users to customize the encoding network for conditional inputs. RNACG exhibits extensive applicability in sequence generation and property prediction tasks.
  arXiv  Detail & Related papers  (2024-07-29T09:46:46Z)
• BEACON: Benchmark for Comprehensive RNA Tasks and Language Models [60.02663015002029]
  We introduce the first comprehensive RNA benchmark BEACON (textbfBEnchmtextbfArk for textbfCOmprehensive RtextbfNA Task and Language Models). First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications. Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models. Third, we investigate the vital RNA language model components
  arXiv  Detail & Related papers  (2024-06-14T19:39:19Z)
• Machine Learning Modeling Of SiRNA Structure-Potency Relationship With Applications Against Sars-Cov-2 Spike Gene [0.0]
  Drug discovery process is lengthy and costly, taking nearly a decade to bring a new drug to the market. Biotechnology, computational methods, and machine learning algorithms have the potential to revolutionize drug discovery, speeding up the process and improving patient outcomes. The COVID-19 pandemic has further accelerated and deepened the recognition of the potential of these techniques, especially in the areas of drug repurposing and efficacy predictions.
  arXiv  Detail & Related papers  (2024-01-18T23:00:34Z)
• scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis in Brain [46.39828178736219]
  We introduce scHyena, a foundation model designed to address these challenges and enhance the accuracy of scRNA-seq analysis in the brain. scHyena is equipped with a linear adaptor layer, the positional encoding via gene-embedding, and a bidirectional Hyena operator. This enables us to process full-length scRNA-seq data without losing any information from the raw data.
  arXiv  Detail & Related papers  (2023-10-04T10:30:08Z)
• Knowledge from Large-Scale Protein Contact Prediction Models Can Be Transferred to the Data-Scarce RNA Contact Prediction Task [40.051834115537474]
  We find that a protein-coevolution Transformer-based deep neural network can be transferred to the RNA contact prediction task. Experiments confirm that RNA contact prediction through transfer learning is greatly improved. Our findings indicate that the learned structural patterns of proteins can be transferred to RNAs, opening up potential new avenues for research.
  arXiv  Detail & Related papers  (2023-02-13T06:00:56Z)
• RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design [65.41144149958208]
  This study aims to systematically construct a data-driven RNA design pipeline. We crafted a benchmark dataset and designed a comprehensive structural modeling approach to represent the complex RNA tertiary structure. We incorporated extracted secondary structures with base pairs as prior knowledge to facilitate the RNA design process.
  arXiv  Detail & Related papers  (2023-01-25T17:19:49Z)
• Retrieval-based Controllable Molecule Generation [63.44583084888342]
  We propose a new retrieval-based framework for controllable molecule generation. We use a small set of molecules to steer the pre-trained generative model towards synthesizing molecules that satisfy the given design criteria. Our approach is agnostic to the choice of generative models and requires no task-specific fine-tuning.
  arXiv  Detail & Related papers  (2022-08-23T17:01:16Z)
• E2Efold-3D: End-to-End Deep Learning Method for accurate de novo RNA 3D Structure Prediction [46.38735421190187]
  We develop the first end-to-end deep learning approach, E2Efold-3D, to accurately perform the textitde novo RNA structure prediction. Several novel components are proposed to overcome the data scarcity, such as a fully-differentiable end-to-end pipeline, secondary structure-assisted self-distillation, and parameter-efficient backbone formulation.
  arXiv  Detail & Related papers  (2022-07-04T17:15:35Z)
• Improving RNA Secondary Structure Design using Deep Reinforcement Learning [69.63971634605797]
  We propose a new benchmark of applying reinforcement learning to RNA sequence design, in which the objective function is defined to be the free energy in the sequence's secondary structure. We show results of the ablation analysis that we do for these algorithms, as well as graphs indicating the algorithm's performance across batches.
  arXiv  Detail & Related papers  (2021-11-05T02:54:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.