A single-cell gene expression language model

• URL: http://arxiv.org/abs/2210.14330v1
• Date: Tue, 25 Oct 2022 20:52:19 GMT
• Title: A single-cell gene expression language model
• Authors: William Connell, Umair Khan, Michael J. Keiser
• Abstract summary: We propose a machine learning system to learn context dependencies between genes. Our model, Exceiver, is trained across a diversity of cell types using a self-supervised task. We found agreement between the similarity profiles of latent sample representations and learned gene embeddings with respect to biological annotations.
• Score: 2.9112649816695213
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gene regulation is a dynamic process that connects genotype and phenotype. Given the difficulty of physically mapping mammalian gene circuitry, we require new computational methods to learn regulatory rules. Natural language is a valuable analogy to the communication of regulatory control. Machine learning systems model natural language by explicitly learning context dependencies between words. We propose a similar system applied to single-cell RNA expression profiles to learn context dependencies between genes. Our model, Exceiver, is trained across a diversity of cell types using a self-supervised task formulated for discrete count data, accounting for feature sparsity. We found agreement between the similarity profiles of latent sample representations and learned gene embeddings with respect to biological annotations. We evaluated Exceiver on a new dataset and a downstream prediction task and found that pretraining supports transfer learning. Our work provides a framework to model gene regulation on a single-cell level and transfer knowledge to downstream tasks.

Related papers

• Long-range gene expression prediction with token alignment of large language model [37.10820914895689]
  We introduce Genetic sequence Token Alignment (GTA), which aligns genetic sequence features with natural language tokens. GTA learns the regulatory grammar and allows us to further incorporate gene-specific human annotations as prompts. GTA represents a powerful and novel cross-modal approach to gene expression prediction by utilizing a pretrained language model.
  arXiv  Detail & Related papers  (2024-10-02T02:42:29Z)
• Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification [119.13058298388101]
  We develop a Biological-knowledge enhanced PathGenomic multi-label Transformer to improve genetic mutation prediction performances. BPGT first establishes a novel gene encoder that constructs gene priors by two carefully designed modules. BPGT then designs a label decoder that finally performs genetic mutation prediction by two tailored modules.
  arXiv  Detail & Related papers  (2024-06-05T06:42:27Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Data [22.938437500266847]
  We introduce a novel model called Multimodal Similarity Learning Graph Neural Network. It combines Multimodal Machine Learning and Deep Graph Neural Networks to learn gene representations from single-cell sequencing and spatial transcriptomic data. Our model efficiently produces unified gene representations for the analysis of gene functions, tissue functions, diseases, and species evolution.
  arXiv  Detail & Related papers  (2023-09-29T13:33:53Z)
• Neural network facilitated ab initio derivation of linear formula: A case study on formulating the relationship between DNA motifs and gene expression [8.794181445664243]
  We propose a framework for ab initio derivation of sequence motifs and linear formula using a new approach based on the interpretable neural network model. We showed that this linear model could predict gene expression levels using promoter sequences with a performance comparable to deep neural network models.
  arXiv  Detail & Related papers  (2022-08-19T22:29:30Z)
• SemanticCAP: Chromatin Accessibility Prediction Enhanced by Features Learning from a Language Model [3.0643865202019698]
  We propose a new solution named SemanticCAP to identify accessible regions of the genome. It introduces a gene language model which models the context of gene sequences, thus being able to provide an effective representation of gene sequences. Compared with other systems under public benchmarks, our model proved to have better performance.
  arXiv  Detail & Related papers  (2022-04-05T11:47:58Z)
• Epigenomic language models powered by Cerebras [0.0]
  Epigenomic BERT (or EBERT) learns representations based on both DNA sequence and paired epigenetic state inputs. We show EBERT's transfer learning potential by demonstrating strong performance on a cell type-specific transcription factor binding prediction task. Our fine-tuned model exceeds state of the art performance on 4 of 13 evaluation datasets from ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge leaderboard.
  arXiv  Detail & Related papers  (2021-12-14T17:23:42Z)
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types [75.65676405302105]
  We propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences.
  arXiv  Detail & Related papers  (2021-10-11T12:48:44Z)
• Reprogramming Language Models for Molecular Representation Learning [65.00999660425731]
  We propose Representation Reprogramming via Dictionary Learning (R2DL) for adversarially reprogramming pretrained language models for molecular learning tasks. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver. R2DL achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data and outperforms the baseline in a limited training-data setting.
  arXiv  Detail & Related papers  (2020-12-07T05:50:27Z)
• Select-ProtoNet: Learning to Select for Few-Shot Disease Subtype Prediction [55.94378672172967]
  We focus on few-shot disease subtype prediction problem, identifying subgroups of similar patients. We introduce meta learning techniques to develop a new model, which can extract the common experience or knowledge from interrelated clinical tasks. Our new model is built upon a carefully designed meta-learner, called Prototypical Network, that is a simple yet effective meta learning machine for few-shot image classification.
  arXiv  Detail & Related papers  (2020-09-02T02:50:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.