Transformer-Based Representation Learning for Robust Gene Expression Modeling and Cancer Prognosis

• URL: http://arxiv.org/abs/2504.09704v1
• Date: Sun, 13 Apr 2025 19:49:59 GMT
• Title: Transformer-Based Representation Learning for Robust Gene Expression Modeling and Cancer Prognosis
• Authors: Shuai Jiang, Saeed Hassanpour,
• Abstract summary: We present GexBERT, a transformer-based autoencoder framework for robust representation learning of gene expression data.<n>GexBERT learns context-aware gene embeddings by pretraining on large-scale transcriptomic profiles.<n>It achieves state-of-the-art classification accuracy from limited gene subsets, improves survival prediction by restoring expression of prognostic anchor genes, and outperforms conventional imputation methods under high missingness.
• Score: 3.782770832189636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based models have achieved remarkable success in natural language and vision tasks, but their application to gene expression analysis remains limited due to data sparsity, high dimensionality, and missing values. We present GexBERT, a transformer-based autoencoder framework for robust representation learning of gene expression data. GexBERT learns context-aware gene embeddings by pretraining on large-scale transcriptomic profiles with a masking and restoration objective that captures co-expression relationships among thousands of genes. We evaluate GexBERT across three critical tasks in cancer research: pan-cancer classification, cancer-specific survival prediction, and missing value imputation. GexBERT achieves state-of-the-art classification accuracy from limited gene subsets, improves survival prediction by restoring expression of prognostic anchor genes, and outperforms conventional imputation methods under high missingness. Furthermore, its attention-based interpretability reveals biologically meaningful gene patterns across cancer types. These findings demonstrate the utility of GexBERT as a scalable and effective tool for gene expression modeling, with translational potential in settings where gene coverage is limited or incomplete.

Related papers

• Learning to Discover Regulatory Elements for Gene Expression Prediction [59.470991831978516]
  Seq2Exp is a Sequence to Expression network designed to discover and extract regulatory elements that drive target gene expression. Our approach captures the causal relationship between epigenomic signals, DNA sequences and their associated regulatory elements.
  arXiv  Detail & Related papers  (2025-02-19T03:25:49Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Precision Cancer Classification and Biomarker Identification from mRNA Gene Expression via Dimensionality Reduction and Explainable AI [0.9423257767158634]
  This research presents a comprehensive pipeline designed to accurately identify 33 distinct cancer types and their corresponding gene sets. It incorporates a combination of normalization and feature selection techniques to reduce dataset dimensionality effectively. We leverage Explainable AI to elucidate the biological significance of the identified cancer-specific genes.
  arXiv  Detail & Related papers  (2024-10-08T18:56:31Z)
• Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification [119.13058298388101]
  We develop a Biological-knowledge enhanced PathGenomic multi-label Transformer to improve genetic mutation prediction performances. BPGT first establishes a novel gene encoder that constructs gene priors by two carefully designed modules. BPGT then designs a label decoder that finally performs genetic mutation prediction by two tailored modules.
  arXiv  Detail & Related papers  (2024-06-05T06:42:27Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• Accurate Spatial Gene Expression Prediction by integrating Multi-resolution features [0.0]
  TRIPLEX is a novel deep learning framework designed to predict spatial gene expression from Whole Slide Images (WSIs) Our benchmark study, conducted on three public ST datasets, demonstrates that TRIPLEX outperforms current state-of-the-art models in Mean Squared Error (MSE), Mean Absolute Error (MAE), and Pearson Correlation Coefficient (PCC) The model's predictions align closely with ground truth gene expression profiles and tumor annotations, underscoring TRIPLEX's potential in advancing cancer diagnosis and treatment.
  arXiv  Detail & Related papers  (2024-03-12T12:25:38Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• Breast Cancer Histopathology Image based Gene Expression Prediction using Spatial Transcriptomics data and Deep Learning [3.583756449759971]
  We present BrST-Net, a deep learning framework for predicting gene expression from histopathology images. We trained and evaluated 10 state-of-the-art deep learning models without utilizing pretrained weights for the prediction of 250 genes. Our methodology outperforms previous studies, with 237 genes identified with positive correlation, including 24 genes with a median correlation coefficient greater than 0.50.
  arXiv  Detail & Related papers  (2023-03-17T14:03:40Z)
• Machine Learning Methods for Cancer Classification Using Gene Expression Data: A Review [77.34726150561087]
  Cancer is the second major cause of death after cardiovascular diseases. Gene expression can play a fundamental role in the early detection of cancer. This study reviews recent progress in gene expression analysis for cancer classification using machine learning methods.
  arXiv  Detail & Related papers  (2023-01-28T15:03:03Z)
• Attention-based Interpretable Regression of Gene Expression in Histology [0.0]
  Interpretability of deep learning is widely used to evaluate the reliability of medical imaging models. We show that interpretability can reveal connections between the microscopic appearance of cancer tissue and its gene expression profiling.
  arXiv  Detail & Related papers  (2022-08-29T07:30:33Z)
• Low-Rank Reorganization via Proportional Hazards Non-negative Matrix Factorization Unveils Survival Associated Gene Clusters [9.773075235189525]
  In this work, Cox proportional hazards regression is integrated with NMF by imposing survival constraints. Using human cancer gene expression data, the proposed technique can unravel critical clusters of cancer genes. The discovered gene clusters reflect rich biological implications and can help identify survival-related biomarkers.
  arXiv  Detail & Related papers  (2020-08-09T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.