Can sparse autoencoders make sense of latent representations?

• URL: http://arxiv.org/abs/2410.11468v2
• Date: Mon, 03 Feb 2025 18:20:35 GMT
• Title: Can sparse autoencoders make sense of latent representations?
• Authors: Viktoria Schuster,
• Abstract summary: We find that latent representations can encode observable and directly connected hidden variables in superposition.<n>Applying to single-cell multi-omics data, we show that SAEs can uncover key biological processes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse autoencoders (SAEs) have lately been used to uncover interpretable latent features in large language models. Here, we explore their potential for decomposing latent representations in complex and high-dimensional biological data, where the underlying variables are often unknown. Using simulated data, we find that latent representations can encode observable and directly connected upstream hidden variables in superposition. The degree to which they are learned depends on the type of variable and the model architecture, favoring shallow and wide networks. Superpositions, however, are not identifiable if the generative variables are unknown. SAEs can recover these variables and their structure with respect to the observables. Applied to single-cell multi-omics data, we show that SAEs can uncover key biological processes. We further present an automated method for linking SAE features to biological concepts to enable large-scale analysis of single-cell expression models.

Related papers

• Dense SAE Latents Are Features, Not Bugs [75.08462524662072]
  We show that dense latents serve functional roles in language model computation.<n>We identify classes tied to position tracking, context binding, entropy regulation, letter-specific output signals, part-of-speech, and principal component reconstruction.
  arXiv  Detail & Related papers  (2025-06-18T17:59:35Z)
• CellVerse: Do Large Language Models Really Understand Cell Biology? [74.34984441715517]
  We introduce CellVerse, a unified language-centric question-answering benchmark that integrates four types of single-cell multi-omics data.<n>We systematically evaluate the performance across 14 open-source and closed-source LLMs ranging from 160M to 671B on CellVerse.
  arXiv  Detail & Related papers  (2025-05-09T06:47:23Z)
• GRAPE: Heterogeneous Graph Representation Learning for Genetic Perturbation with Coding and Non-Coding Biotype [51.58774936662233]
  Building gene regulatory networks (GRN) is essential to understand and predict the effects of genetic perturbations.<n>In this work, we leverage pre-trained large language model and DNA sequence model to extract features from gene descriptions and DNA sequence data.<n>We introduce gene biotype information for the first time in genetic perturbation, simulating the distinct roles of genes with different biotypes in regulating cellular processes.
  arXiv  Detail & Related papers  (2025-05-06T03:35:24Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model [76.95536611263356]
  PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy.
  arXiv  Detail & Related papers  (2024-12-17T09:59:53Z)
• States Hidden in Hidden States: LLMs Emerge Discrete State Representations Implicitly [72.24742240125369]
  In this paper, we uncover the intrinsic ability to perform extended sequences of calculations without relying on chain-of-thought step-by-step solutions. Remarkably, the most advanced models can directly output the results of two-digit number additions with lengths extending up to 15 addends.
  arXiv  Detail & Related papers  (2024-07-16T06:27:22Z)
• Multimodal contrastive learning for spatial gene expression prediction using histology images [13.47034080678041]
  We propose textbfmclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. textbfmclSTExp has superior performance in predicting spatial gene expression. It has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists.
  arXiv  Detail & Related papers  (2024-07-11T06:33:38Z)
• Semantically Rich Local Dataset Generation for Explainable AI in Genomics [0.716879432974126]
  Black box deep learning models trained on genomic sequences excel at predicting the outcomes of different gene regulatory mechanisms. We propose using Genetic Programming to generate datasets by evolving perturbations in sequences that contribute to their semantic diversity.
  arXiv  Detail & Related papers  (2024-07-03T10:31:30Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• Tertiary Lymphoid Structures Generation through Graph-based Diffusion [54.37503714313661]
  In this work, we leverage state-of-the-art graph-based diffusion models to generate biologically meaningful cell-graphs. We show that the adopted graph diffusion model is able to accurately learn the distribution of cells in terms of their tertiary lymphoid structures (TLS) content.
  arXiv  Detail & Related papers  (2023-10-10T14:37:17Z)
• Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction [3.2274401541163322]
  We propose a memory-efficient multimodal Transformer that can model interactions between pathway and histology patch tokens. Our proposed model, SURVPATH, achieves state-of-the-art performance when evaluated against both unimodal and multimodal baselines.
  arXiv  Detail & Related papers  (2023-04-13T21:02:32Z)
• Posterior Collapse and Latent Variable Non-identifiability [54.842098835445]
  We propose a class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility. Across synthetic and real datasets, latent-identifiable variational autoencoders outperform existing methods in mitigating posterior collapse and providing meaningful representations of the data.
  arXiv  Detail & Related papers  (2023-01-02T06:16:56Z)
• Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling [3.2435888122704037]
  We propose a deep generative model of single-cell gene expression data for which each perturbation is treated as an intervention targeting an unknown, but sparse, subset of latent variables. We benchmark these methods on simulated single-cell data to evaluate their performance at latent units recovery, causal target identification and out-of-domain generalization.
  arXiv  Detail & Related papers  (2022-11-07T15:47:40Z)
• Modelling Technical and Biological Effects in scRNA-seq data with Scalable GPLVMs [6.708052194104378]
  We extend a popular approach for probabilistic non-linear dimensionality reduction, the Gaussian process latent variable model, to scale to massive single-cell datasets. The key idea is to use an augmented kernel which preserves the factorisability of the lower bound allowing for fast variational inference.
  arXiv  Detail & Related papers  (2022-09-14T15:25:15Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• SubOmiEmbed: Self-supervised Representation Learning of Multi-omics Data for Cancer Type Classification [4.992154875028543]
  Integration and analysis of multi-omics data give us a broad view of tumours, which can improve clinical decision making. SubOmiEmbed produces comparable results to the baseline OmiEmbed with a much smaller network and by using just a subset of the data. This work can be improved to integrate mutation-based genomic data as well.
  arXiv  Detail & Related papers  (2022-02-03T16:39:09Z)
• MURAL: An Unsupervised Random Forest-Based Embedding for Electronic Health Record Data [59.26381272149325]
  We present an unsupervised random forest for representing data with disparate variable types. MURAL forests consist of a set of decision trees where node-splitting variables are chosen at random. We show that using our approach, we can visualize and classify data more accurately than competing approaches.
  arXiv  Detail & Related papers  (2021-11-19T22:02:21Z)
• Identifiable Variational Autoencoders via Sparse Decoding [37.30831737046145]
  We develop the Sparse VAE, a deep generative model for unsupervised representation learning on high-dimensional data. We first show that the Sparse VAE is identifiable: given data drawn from the model, there exists a uniquely optimal set of factors. We empirically study the Sparse VAE with both simulated and real data.
  arXiv  Detail & Related papers  (2021-10-20T22:11:33Z)
• Encoding Domain Information with Sparse Priors for Inferring Explainable Latent Variables [2.8935588665357077]
  We propose spex-LVM, a factorial latent variable model with sparse priors to encourage the inference of explainable factors. spex-LVM utilizes existing knowledge of curated biomedical pathways to automatically assign annotated attributes to latent factors. Evaluations on simulated and real single-cell RNA-seq datasets demonstrate that our model robustly identifies relevant structure in an inherently explainable manner.
  arXiv  Detail & Related papers  (2021-07-08T10:19:32Z)
• Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders [63.46738617561255]
  We consider the problem of sparsifying the discrete latent space of a trained conditional variational autoencoder. We use evidential theory to identify the latent classes that receive direct evidence from a particular input condition and filter out those that do not. Experiments on diverse tasks, such as image generation and human behavior prediction, demonstrate the effectiveness of our proposed technique.
  arXiv  Detail & Related papers  (2020-10-19T01:27:21Z)
• Category-Learning with Context-Augmented Autoencoder [63.05016513788047]
  Finding an interpretable non-redundant representation of real-world data is one of the key problems in Machine Learning. We propose a novel method of using data augmentations when training autoencoders. We train a Variational Autoencoder in such a way, that it makes transformation outcome predictable by auxiliary network.
  arXiv  Detail & Related papers  (2020-10-10T14:04:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.