Long-range gene expression prediction with token alignment of large language model

• URL: http://arxiv.org/abs/2410.01858v1
• Date: Wed, 2 Oct 2024 02:42:29 GMT
• Title: Long-range gene expression prediction with token alignment of large language model
• Authors: Edouardo Honig, Huixin Zhan, Ying Nian Wu, Zijun Frank Zhang,
• Abstract summary: We introduce Genetic sequence Token Alignment (GTA), which aligns genetic sequence features with natural language tokens. GTA learns the regulatory grammar and allows us to further incorporate gene-specific human annotations as prompts. GTA represents a powerful and novel cross-modal approach to gene expression prediction by utilizing a pretrained language model.
• Score: 37.10820914895689
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gene expression is a cellular process that plays a fundamental role in human phenotypical variations and diseases. Despite advances of deep learning models for gene expression prediction, recent benchmarks have revealed their inability to learn distal regulatory grammar. Here, we address this challenge by leveraging a pretrained large language model to enhance gene expression prediction. We introduce Genetic sequence Token Alignment (GTA), which aligns genetic sequence features with natural language tokens, allowing for symbolic reasoning of genomic sequence features via the frozen language model. This cross-modal adaptation learns the regulatory grammar and allows us to further incorporate gene-specific human annotations as prompts, enabling in-context learning that is not possible with existing models. Trained on lymphoblastoid cells, GTA was evaluated on cells from the Geuvadis consortium and outperforms state-of-the-art models such as Enformer, achieving a Spearman correlation of 0.65, a 10\% improvement. Additionally, GTA offers improved interpretation of long-range interactions through the identification of the most meaningful sections of the input genetic context. GTA represents a powerful and novel cross-modal approach to gene expression prediction by utilizing a pretrained language model, in a paradigm shift from conventional gene expression models trained only on sequence data.

Related papers

• Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification [119.13058298388101]
  We develop a Biological-knowledge enhanced PathGenomic multi-label Transformer to improve genetic mutation prediction performances. BPGT first establishes a novel gene encoder that constructs gene priors by two carefully designed modules. BPGT then designs a label decoder that finally performs genetic mutation prediction by two tailored modules.
  arXiv  Detail & Related papers  (2024-06-05T06:42:27Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• Machine Learning Methods for Cancer Classification Using Gene Expression Data: A Review [77.34726150561087]
  Cancer is the second major cause of death after cardiovascular diseases. Gene expression can play a fundamental role in the early detection of cancer. This study reviews recent progress in gene expression analysis for cancer classification using machine learning methods.
  arXiv  Detail & Related papers  (2023-01-28T15:03:03Z)
• A single-cell gene expression language model [2.9112649816695213]
  We propose a machine learning system to learn context dependencies between genes. Our model, Exceiver, is trained across a diversity of cell types using a self-supervised task. We found agreement between the similarity profiles of latent sample representations and learned gene embeddings with respect to biological annotations.
  arXiv  Detail & Related papers  (2022-10-25T20:52:19Z)
• Granger causal inference on DAGs identifies genomic loci regulating transcription [77.58911272503771]
  GrID-Net is a framework based on graph neural networks with lagged message passing for Granger causal inference on DAG-structured systems. Our application is the analysis of single-cell multimodal data to identify genomic loci that mediate the regulation of specific genes.
  arXiv  Detail & Related papers  (2022-10-18T21:15:10Z)
• Neural network facilitated ab initio derivation of linear formula: A case study on formulating the relationship between DNA motifs and gene expression [8.794181445664243]
  We propose a framework for ab initio derivation of sequence motifs and linear formula using a new approach based on the interpretable neural network model. We showed that this linear model could predict gene expression levels using promoter sequences with a performance comparable to deep neural network models.
  arXiv  Detail & Related papers  (2022-08-19T22:29:30Z)
• Epigenomic language models powered by Cerebras [0.0]
  Epigenomic BERT (or EBERT) learns representations based on both DNA sequence and paired epigenetic state inputs. We show EBERT's transfer learning potential by demonstrating strong performance on a cell type-specific transcription factor binding prediction task. Our fine-tuned model exceeds state of the art performance on 4 of 13 evaluation datasets from ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge leaderboard.
  arXiv  Detail & Related papers  (2021-12-14T17:23:42Z)
• All You Need is Color: Image based Spatial Gene Expression Prediction using Neural Stain Learning [11.9045433112067]
  We propose a "stain-aware" machine learning approach for prediction of spatial transcriptomic gene expression profiles. We have found that the gene expression predictions from the proposed approach show higher correlations with true expression values obtained through sequencing.
  arXiv  Detail & Related papers  (2021-08-23T23:43:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.