White-Box Diffusion Transformer for single-cell RNA-seq generation

• URL: http://arxiv.org/abs/2411.06785v2
• Date: Sat, 16 Nov 2024 08:36:17 GMT
• Title: White-Box Diffusion Transformer for single-cell RNA-seq generation
• Authors: Zhuorui Cui, Shengze Dong, Ding Liu,
• Abstract summary: We propose a hybrid model based on Diffusion model and White-Box transformer to generate synthetic and biologically plausible scRNA-seq data. Our White-Box Diffusion Transformer combines the generative capabilities of Diffusion model with the mathematical interpretability of White-Box transformer.
• Score: 9.846966401472802
• License:
• Abstract: As a powerful tool for characterizing cellular subpopulations and cellular heterogeneity, single cell RNA sequencing (scRNA-seq) technology offers advantages of high throughput and multidimensional analysis. However, the process of data acquisition is often constrained by high cost and limited sample availability. To overcome these limitations, we propose a hybrid model based on Diffusion model and White-Box transformer that aims to generate synthetic and biologically plausible scRNA-seq data. Diffusion model progressively introduce noise into the data and then recover the original data through a denoising process, a forward and reverse process that is particularly suitable for generating complex data distributions. White-Box transformer is a deep learning architecture that emphasizes mathematical interpretability. By minimizing the encoding rate of the data and maximizing the sparsity of the representation, it not only reduces the computational burden, but also provides clear insight into underlying structure. Our White-Box Diffusion Transformer combines the generative capabilities of Diffusion model with the mathematical interpretability of White-Box transformer. Through experiments using six different single-cell RNA-Seq datasets, we visualize both generated and real data using t-SNE dimensionality reduction technique, as well as quantify similarity between generated and real data using various metrics to demonstrate comparable performance of White-Box Diffusion Transformer and Diffusion Transformer in generating scRNA-seq data alongside significant improvements in training efficiency and resource utilization. Our code is available at https://github.com/lingximamo/White-Box-Diffusion-Transformer

Related papers

• Variable-size Symmetry-based Graph Fourier Transforms for image compression [65.7352685872625]
  We propose a new family of Symmetry-based Graph Fourier Transforms of variable sizes into a coding framework. Our proposed algorithm generates symmetric graphs on the grid by adding specific symmetrical connections between nodes. Experiments show that SBGFTs outperform the primary transforms integrated in the explicit Multiple Transform Selection.
  arXiv  Detail & Related papers  (2024-11-24T13:00:44Z)
• Scalable Amortized GPLVMs for Single Cell Transcriptomics Data [9.010523724015398]
  Dimensionality reduction is crucial for analyzing large-scale single-cell RNA-seq data. We introduce an improved model, the amortized variational model (BGPLVM) BGPLVM is tailored for single-cell RNA-seq with specialized encoder, kernel, and likelihood designs.
  arXiv  Detail & Related papers  (2024-05-06T21:54:38Z)
• scRDiT: Generating single-cell RNA-seq data by diffusion transformers and accelerating sampling [9.013834280011293]
  Single-cell RNA sequencing (scRNA-seq) is a groundbreaking technology extensively utilized in biological research. Our study introduces a generative approach termed scRNA-seq Diffusion Transformer (scRDiT) This method generates virtual scRNA-seq data by leveraging a real dataset.
  arXiv  Detail & Related papers  (2024-04-09T09:25:16Z)
• scDiffusion: conditional generation of high-quality single-cell data using diffusion model [1.0738561302102216]
  Single-cell RNA sequencing (scRNA-seq) data are important for studying the laws of life at single-cell level. It is still challenging to obtain enough high-quality scRNA-seq data. We developed scDiffusion, a generative model combining diffusion model and foundation model to generate high-quality scRNA-seq data.
  arXiv  Detail & Related papers  (2024-01-08T15:44:39Z)
• xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data [45.043516102428676]
  We propose a novel asymmetric encoder-decoder transformer for scRNA-seq data, called xTrimoGene$alpha$ (or xTrimoGene for short) xTrimoGene reduces FLOPs by one to two orders of magnitude compared to classical transformers while maintaining high accuracy. Our experiments also show that the performance of xTrimoGene improves as we scale up the model sizes.
  arXiv  Detail & Related papers  (2023-11-26T01:23:01Z)
• scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis in Brain [46.39828178736219]
  We introduce scHyena, a foundation model designed to address these challenges and enhance the accuracy of scRNA-seq analysis in the brain. scHyena is equipped with a linear adaptor layer, the positional encoding via gene-embedding, and a bidirectional Hyena operator. This enables us to process full-length scRNA-seq data without losing any information from the raw data.
  arXiv  Detail & Related papers  (2023-10-04T10:30:08Z)
• Diffusion-GAN: Training GANs with Diffusion [135.24433011977874]
  Generative adversarial networks (GANs) are challenging to train stably. We propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate instance noise. We show that Diffusion-GAN can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.
  arXiv  Detail & Related papers  (2022-06-05T20:45:01Z)
• Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration [77.99182201815763]
  Leveraging prior knowledge on intraclass variance due to transformations is a powerful method to improve the sample complexity of deep neural networks. We propose a novel monomial selection algorithm based on pruning methods to allow an application to more complex problems. We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and CIFAR-10 datasets.
  arXiv  Detail & Related papers  (2022-02-08T16:16:11Z)
• TransCMD: Cross-Modal Decoder Equipped with Transformer for RGB-D Salient Object Detection [86.94578023985677]
  In this work, we rethink this task from the perspective of global information alignment and transformation. Specifically, the proposed method (TransCMD) cascades several cross-modal integration units to construct a top-down transformer-based information propagation path. Experimental results on seven RGB-D SOD benchmark datasets demonstrate that a simple two-stream encoder-decoder framework can surpass the state-of-the-art purely CNN-based methods.
  arXiv  Detail & Related papers  (2021-12-04T15:45:34Z)
• nnFormer: Interleaved Transformer for Volumetric Segmentation [50.10441845967601]
  We introduce nnFormer, a powerful segmentation model with an interleaved architecture based on empirical combination of self-attention and convolution. nnFormer achieves tremendous improvements over previous transformer-based methods on two commonly used datasets Synapse and ACDC.
  arXiv  Detail & Related papers  (2021-09-07T17:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.