Transformer-based Single-Cell Language Model: A Survey

• URL: http://arxiv.org/abs/2407.13205v1
• Date: Thu, 18 Jul 2024 06:43:12 GMT
• Title: Transformer-based Single-Cell Language Model: A Survey
• Abstract summary: We provide a detailed introduction about the structure and principles of transformers. We review the single-cell language models and large language models for single-cell data analysis. We discuss the challenges of single-cell language models and provide promising research directions.
• Score: 5.228439173541588
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The transformers have achieved significant accomplishments in the natural language processing as its outstanding parallel processing capabilities and highly flexible attention mechanism. In addition, increasing studies based on transformers have been proposed to model single-cell data. In this review, we attempt to systematically summarize the single-cell language models and applications based on transformers. First, we provide a detailed introduction about the structure and principles of transformers. Then, we review the single-cell language models and large language models for single-cell data analysis. Moreover, we explore the datasets and applications of single-cell language models in downstream tasks such as batch correction, cell clustering, cell type annotation, gene regulatory network inference and perturbation response. Further, we discuss the challenges of single-cell language models and provide promising research directions. We hope this review will serve as an up-to-date reference for researchers interested in the direction of single-cell language models.

Related papers

• Prototype Guided Post-pretraining for Single-Cell Representation Learning [0.0]
  Single-cell representation learning from gene expression data offers a way to uncover the complex regulatory logic underlying cellular function.<n>Several single-cell pretrained models have recently been proposed that treat genes as tokens and cells as sentences.<n>We introduce CellRefine, a post-pretraining method that operates between the pretraining and fine-tuning stages of a single-cell foundation model.
  arXiv  Detail & Related papers  (2026-05-08T16:08:10Z)
• Lingshu-Cell: A generative cellular world model for transcriptome modeling toward virtual cells [48.885716676667776]
  Lingshu-Cell is a discrete diffusion model that learns transcriptomic state distributions and supports conditional simulation under perturbations.<n>It captures complex transcriptome-wide expression dependencies across approximately 18,000 genes without relying on prior gene selection.<n>It achieves leading performance on the Virtual Cell Challenge H1 genetic perturbation benchmark and in predicting cytokine-induced responses in human PBMCs.
  arXiv  Detail & Related papers  (2026-03-26T09:46:27Z)
• Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training [7.812507078660317]
  We propose an Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model (OKR-CELL)<n>It is built based on a cross-modal Cell-Language pre-training framework, which comprises two key innovations.<n>OKR-CELL obtains cutting-edge results across 6 evaluation tasks.
  arXiv  Detail & Related papers  (2026-01-09T09:10:14Z)
• Large Language Models Meet Virtual Cell: A Survey [5.3507304878218145]
  Large language models (LLMs) are transforming cellular biology by enabling the development of "virtual cells"<n>This work provides a comprehensive review of LLMs for virtual cell modeling.
  arXiv  Detail & Related papers  (2025-10-09T02:41:30Z)
• CellForge: Agentic Design of Virtual Cell Models [24.938939602572702]
  We introduce CellForge, an agentic system that transforms presented biological datasets into optimized computational models for virtual cells.<n>The framework integrates three core modules: Task Analysis, Method Design, and Experiment Execution.<n>We demonstrate CellForge's capabilities in single-cell perturbation prediction, using six diverse datasets.
  arXiv  Detail & Related papers  (2025-08-04T10:43:31Z)
• CellVerse: Do Large Language Models Really Understand Cell Biology? [74.34984441715517]
  We introduce CellVerse, a unified language-centric question-answering benchmark that integrates four types of single-cell multi-omics data.<n>We systematically evaluate the performance across 14 open-source and closed-source LLMs ranging from 160M to 671B on CellVerse.
  arXiv  Detail & Related papers  (2025-05-09T06:47:23Z)
• A scalable gene network model of regulatory dynamics in single cells [88.48246132084441]
  We introduce a Functional Learnable model of Cell dynamicS, FLeCS, that incorporates gene network structure into coupled differential equations to model gene regulatory functions. Given (pseudo)time-series single-cell data, FLeCS accurately infers cell dynamics at scale.
  arXiv  Detail & Related papers  (2025-03-25T19:19:21Z)
• A Multi-Modal AI Copilot for Single-Cell Analysis with Instruction Following [32.67347401145835]
  Large language models excel at interpreting complex natural language instructions, enabling them to perform a wide range of tasks. We present InstructCell, a multi-modal AI copilot that leverages natural language as a medium for more direct and flexible single-cell analysis. InstructCell empowers researchers to accomplish critical tasks-such as cell type annotation, conditional pseudo-cell generation, and drug sensitivity prediction-using straightforward natural language commands.
  arXiv  Detail & Related papers  (2025-01-14T15:12:19Z)
• scReader: Prompting Large Language Models to Interpret scRNA-seq Data [12.767105992391555]
  We propose an innovative hybrid approach that integrates the general knowledge capabilities of large language models with domain-specific representation models for single-cell omics data interpretation. By inputting single-cell gene-level expression data with prompts, we effectively model cellular representations based on the differential expression levels of genes across various species and cell types.
  arXiv  Detail & Related papers  (2024-12-24T04:28:42Z)
• Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data [13.56585855722118]
  Large language models (LLMs) have demonstrated their ability to efficiently process and synthesize vast corpora of text to automatically extract biological knowledge. Our study explores the potential of LLMs to accurately classify and annotate cell types in single-cell RNA sequencing (scRNA-seq) data. The results demonstrate that LLMs can provide robust interpretations of single-cell data without requiring additional fine-tuning.
  arXiv  Detail & Related papers  (2024-12-03T23:58:35Z)
• Multi-Modal and Multi-Attribute Generation of Single Cells with CFGen [76.02070962797794]
  This work introduces CellFlow for Generation (CFGen), a flow-based conditional generative model that preserves the inherent discreteness of single-cell data. CFGen generates whole-genome multi-modal single-cell data reliably, improving the recovery of crucial biological data characteristics.
  arXiv  Detail & Related papers  (2024-07-16T14:05:03Z)
• LangCell: Language-Cell Pre-training for Cell Identity Understanding [3.6518971609937068]
  We introduce LangCell, a unified representation of single-cell data and natural language during the pre-training phase. Results show that LangCell is the only single-cell PLM that can work effectively in zero-shot cell identity understanding scenarios.
  arXiv  Detail & Related papers  (2024-05-09T10:04:05Z)
• Scalable Amortized GPLVMs for Single Cell Transcriptomics Data [9.010523724015398]
  Dimensionality reduction is crucial for analyzing large-scale single-cell RNA-seq data. We introduce an improved model, the amortized variational model (BGPLVM) BGPLVM is tailored for single-cell RNA-seq with specialized encoder, kernel, and likelihood designs.
  arXiv  Detail & Related papers  (2024-05-06T21:54:38Z)
• scInterpreter: Training Large Language Models to Interpret scRNA-seq Data for Cell Type Annotation [15.718901418627366]
  This research focuses on how to train and adapt the Large Language Model with the capability to interpret and distinguish cell types in single-cell RNA sequencing data.
  arXiv  Detail & Related papers  (2024-02-18T05:39:00Z)
• Controllable Topic-Focused Abstractive Summarization [57.8015120583044]
  Controlled abstractive summarization focuses on producing condensed versions of a source article to cover specific aspects. This paper presents a new Transformer-based architecture capable of producing topic-focused summaries.
  arXiv  Detail & Related papers  (2023-11-12T03:51:38Z)
• Mixed Models with Multiple Instance Learning [51.440557223100164]
  We introduce MixMIL, a framework integrating Generalized Linear Mixed Models (GLMM) and Multiple Instance Learning (MIL) Our empirical results reveal that MixMIL outperforms existing MIL models in single-cell datasets.
  arXiv  Detail & Related papers  (2023-11-04T16:42:42Z)
• Causal machine learning for single-cell genomics [94.28105176231739]
  We discuss the application of machine learning techniques to single-cell genomics and their challenges. We first present the model that underlies most of current causal approaches to single-cell biology. We then identify open problems in the application of causal approaches to single-cell data.
  arXiv  Detail & Related papers  (2023-10-23T13:35:24Z)
• Revolutionizing Single Cell Analysis: The Power of Large Language Models for Cell Type Annotation [0.0]
  Large language models such as ChatGPT and New Bing provide accurate annotations of cell types. By using ChatGPT to annotate single cell data, we can relate rare cell type to their function. This can have important applications in understanding cancer progression, mammalian development, and stem cell differentiation.
  arXiv  Detail & Related papers  (2023-04-05T18:45:54Z)
• N-Grammer: Augmenting Transformers with latent n-grams [35.39961549040385]
  We propose a simple yet effective modification to the Transformer architecture inspired by the literature in statistical language modeling, by augmenting the model with n-grams that are constructed from a discrete latent representation of the text sequence. We evaluate our model, the N-Grammer on language modeling on the C4 data-set as well as text classification on the SuperGLUE data-set, and find that it outperforms several strong baselines such as the Transformer and the Primer.
  arXiv  Detail & Related papers  (2022-07-13T17:18:02Z)
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types [75.65676405302105]
  We propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences.
  arXiv  Detail & Related papers  (2021-10-11T12:48:44Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)
• Towards an Automatic Analysis of CHO-K1 Suspension Growth in Microfluidic Single-cell Cultivation [63.94623495501023]
  We propose a novel Machine Learning architecture, which allows us to infuse a neural deep network with human-powered abstraction on the level of data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated.
  arXiv  Detail & Related papers  (2020-10-20T08:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.