Multi-modal Transfer Learning between Biological Foundation Models

• URL: http://arxiv.org/abs/2406.14150v1
• Date: Thu, 20 Jun 2024 09:44:53 GMT
• Title: Multi-modal Transfer Learning between Biological Foundation Models
• Authors: Juan Jose Garau-Luis, Patrick Bordes, Liam Gonzalez, Masa Roller, Bernardo P. de Almeida, Lorenz Hexemer, Christopher Blum, Stefan Laurent, Jan Grzegorzewski, Maren Lang, Thomas Pierrot, Guillaume Richard,
• Abstract summary: We propose a multi-modal-specific model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality encoders. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods. We open-source our model, paving the way for new multi-modal gene expression approaches.
• Score: 2.6545450959042234
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key to understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to a single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specific encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple RNA transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different transcription expression levels across various human tissues. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods and leveraging the use of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pre-training as well as in between modalities. We open-source our model, paving the way for new multi-modal gene expression approaches.

Related papers

• Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification [53.488387420073536]
  Life-Code is a comprehensive framework that spans different biological functions. Life-Code achieves state-of-the-art performance on various tasks across three omics.
  arXiv  Detail & Related papers  (2025-02-11T06:53:59Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters. The model adheres to the central dogma of molecular biology, accurately generating protein-coding sequences. It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of promoter sequences.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Biology Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models [51.316001071698224]
  We introduce Biology-Instructions, the first large-scale multi-omics biological sequences-related instruction-tuning dataset. This dataset can bridge the gap between large language models (LLMs) and complex biological sequences-related tasks. We also develop a strong baseline called ChatMultiOmics with a novel three-stage training pipeline.
  arXiv  Detail & Related papers  (2024-12-26T12:12:23Z)
• BSM: Small but Powerful Biological Sequence Model for Genes and Proteins [6.6055625629542085]
  We introduce BSM, a small but powerful mixed-modal biological sequence foundation model. It is trained on three types of data: RefSeq, Gene Related Sequences, and interleaved biological sequences from the web. It significantly enhances learning efficiency and cross-modal representation, outperforming models trained solely on unimodal data.
  arXiv  Detail & Related papers  (2024-10-15T11:12:28Z)
• A Benchmark Dataset for Multimodal Prediction of Enzymatic Function Coupling DNA Sequences and Natural Language [3.384797724820242]
  Predicting gene function from its DNA sequence is a fundamental challenge in biology. Deep learning models have been proposed to embed DNA sequences and predict their enzymatic function. Much of the scientific community's knowledge of biological function is not represented in categorical labels.
  arXiv  Detail & Related papers  (2024-07-21T19:27:43Z)
• Generating Multi-Modal and Multi-Attribute Single-Cell Counts with CFGen [76.02070962797794]
  We present Cell Flow for Generation, a flow-based conditional generative model for multi-modal single-cell counts. Our results suggest improved recovery of crucial biological data characteristics while accounting for novel generative tasks.
  arXiv  Detail & Related papers  (2024-07-16T14:05:03Z)
• Semantically Rich Local Dataset Generation for Explainable AI in Genomics [0.716879432974126]
  Black box deep learning models trained on genomic sequences excel at predicting the outcomes of different gene regulatory mechanisms. We propose using Genetic Programming to generate datasets by evolving perturbations in sequences that contribute to their semantic diversity.
  arXiv  Detail & Related papers  (2024-07-03T10:31:30Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• Epigenomic language models powered by Cerebras [0.0]
  Epigenomic BERT (or EBERT) learns representations based on both DNA sequence and paired epigenetic state inputs. We show EBERT's transfer learning potential by demonstrating strong performance on a cell type-specific transcription factor binding prediction task. Our fine-tuned model exceeds state of the art performance on 4 of 13 evaluation datasets from ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge leaderboard.
  arXiv  Detail & Related papers  (2021-12-14T17:23:42Z)
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types [75.65676405302105]
  We propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences.
  arXiv  Detail & Related papers  (2021-10-11T12:48:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.