xTrimoGene: An Efficient and Scalable Representation Learner for
Single-Cell RNA-Seq Data
- URL: http://arxiv.org/abs/2311.15156v2
- Date: Sat, 24 Feb 2024 13:03:49 GMT
- Title: xTrimoGene: An Efficient and Scalable Representation Learner for
Single-Cell RNA-Seq Data
- Authors: Jing Gong, Minsheng Hao, Xingyi Cheng, Xin Zeng, Chiming Liu, Jianzhu
Ma, Xuegong Zhang, Taifeng Wang, Le Song
- Abstract summary: We propose a novel asymmetric encoder-decoder transformer for scRNA-seq data, called xTrimoGene$alpha$ (or xTrimoGene for short)
xTrimoGene reduces FLOPs by one to two orders of magnitude compared to classical transformers while maintaining high accuracy.
Our experiments also show that the performance of xTrimoGene improves as we scale up the model sizes.
- Score: 45.043516102428676
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Advances in high-throughput sequencing technology have led to significant
progress in measuring gene expressions at the single-cell level. The amount of
publicly available single-cell RNA-seq (scRNA-seq) data is already surpassing
50M records for humans with each record measuring 20,000 genes. This highlights
the need for unsupervised representation learning to fully ingest these data,
yet classical transformer architectures are prohibitive to train on such data
in terms of both computation and memory. To address this challenge, we propose
a novel asymmetric encoder-decoder transformer for scRNA-seq data, called
xTrimoGene$^\alpha$ (or xTrimoGene for short), which leverages the sparse
characteristic of the data to scale up the pre-training. This scalable design
of xTrimoGene reduces FLOPs by one to two orders of magnitude compared to
classical transformers while maintaining high accuracy, enabling us to train
the largest transformer models over the largest scRNA-seq dataset today. Our
experiments also show that the performance of xTrimoGene improves as we scale
up the model sizes, and it also leads to SOTA performance over various
downstream tasks, such as cell type annotation, perturb-seq effect prediction,
and drug combination prediction. xTrimoGene model is now available for use as a
service via the following link: https://api.biomap.com/xTrimoGene/apply.
Related papers
- White-Box Diffusion Transformer for single-cell RNA-seq generation [9.846966401472802]
We propose a hybrid model based on Diffusion model and White-Box transformer to generate synthetic and biologically plausible scRNA-seq data.
Our White-Box Diffusion Transformer combines the generative capabilities of Diffusion model with the mathematical interpretability of White-Box transformer.
arXiv Detail & Related papers (2024-11-11T08:24:59Z) - VST++: Efficient and Stronger Visual Saliency Transformer [74.26078624363274]
We develop an efficient and stronger VST++ model to explore global long-range dependencies.
We evaluate our model across various transformer-based backbones on RGB, RGB-D, and RGB-T SOD benchmark datasets.
arXiv Detail & Related papers (2023-10-18T05:44:49Z) - scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis
in Brain [46.39828178736219]
We introduce scHyena, a foundation model designed to address these challenges and enhance the accuracy of scRNA-seq analysis in the brain.
scHyena is equipped with a linear adaptor layer, the positional encoding via gene-embedding, and a bidirectional Hyena operator.
This enables us to process full-length scRNA-seq data without losing any information from the raw data.
arXiv Detail & Related papers (2023-10-04T10:30:08Z) - The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease
detection [51.697248252191265]
This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation.
We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare.
Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
arXiv Detail & Related papers (2023-09-13T10:40:41Z) - Optirank: classification for RNA-Seq data with optimal ranking reference
genes [0.0]
We propose a logistic regression model, optirank, which learns simultaneously the parameters of the model and the genes to use as a reference set in the ranking.
We also consider real classification tasks, which present different kinds of distribution shifts between train and test data.
arXiv Detail & Related papers (2023-01-11T10:49:06Z) - Epigenomic language models powered by Cerebras [0.0]
Epigenomic BERT (or EBERT) learns representations based on both DNA sequence and paired epigenetic state inputs.
We show EBERT's transfer learning potential by demonstrating strong performance on a cell type-specific transcription factor binding prediction task.
Our fine-tuned model exceeds state of the art performance on 4 of 13 evaluation datasets from ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge leaderboard.
arXiv Detail & Related papers (2021-12-14T17:23:42Z) - Multi-modal Self-supervised Pre-training for Regulatory Genome Across
Cell Types [75.65676405302105]
We propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT.
We pre-train our model on the ATAC-seq dataset with 17 million genome sequences.
arXiv Detail & Related papers (2021-10-11T12:48:44Z) - Finetuning Pretrained Transformers into RNNs [81.72974646901136]
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation.
A linear-complexity recurrent variant has proven well suited for autoregressive generation.
This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
arXiv Detail & Related papers (2021-03-24T10:50:43Z) - Approximate kNN Classification for Biomedical Data [1.1852406625172218]
Single-cell RNA-seq (scRNA-seq) is an emerging DNA sequencing technology with promising capabilities but significant computational challenges.
We propose the utilization of approximate nearest neighbor search algorithms for the task of kNN classification in scRNA-seq data.
arXiv Detail & Related papers (2020-12-03T18:30:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.