AlignSAE: Concept-Aligned Sparse Autoencoders

• URL: http://arxiv.org/abs/2512.02004v1
• Date: Mon, 01 Dec 2025 18:58:22 GMT
• Title: AlignSAE: Concept-Aligned Sparse Autoencoders
• Abstract summary: We introduce AlignSAE, a method that aligns SAE features with a defined ontology through a "pre-train, then post-train" curriculum.<n>After an initial unsupervised training phase, we apply supervised post-training to bind specific concepts to dedicated latent slots.<n>This separation creates an interpretable interface where specific relations can be inspected and controlled without interference from unrelated features.
• Score: 47.18866175760984
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) encode factual knowledge within hidden parametric spaces that are difficult to inspect or control. While Sparse Autoencoders (SAEs) can decompose hidden activations into more fine-grained, interpretable features, they often struggle to reliably align these features with human-defined concepts, resulting in entangled and distributed feature representations. To address this, we introduce AlignSAE, a method that aligns SAE features with a defined ontology through a "pre-train, then post-train" curriculum. After an initial unsupervised training phase, we apply supervised post-training to bind specific concepts to dedicated latent slots while preserving the remaining capacity for general reconstruction. This separation creates an interpretable interface where specific relations can be inspected and controlled without interference from unrelated features. Empirical results demonstrate that AlignSAE enables precise causal interventions, such as reliable "concept swaps", by targeting single, semantically aligned slots.

Related papers

• REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling [56.888597798431704]
  Open-World Object Detection (OWOD) requires detectors to identify previously unseen objects as unknown and incrementally incorporate them into the set of known categories.<n>Existing frameworks rely heavily on exemplar replay to mitigate catastrophic forgetting, but in some real applications, storing raw data conflicts with data access restrictions and leads to data exposure risks.<n>We propose REAL-OW, a novel rehearsal-free framework that decouples incremental knowledge through a collaborative adapter architecture.
  arXiv  Detail & Related papers  (2026-07-03T06:38:54Z)
• Aligning Sentence Embeddings to Human Concepts via Sparse Autoencoders [5.053354418366577]
  We propose a method to disentangle the dense representations of sentence transformers (e.g., E5) into human-interpretable concepts using Top-k Sparse Autoencoders (SAEs)<n>We demonstrate that these disentangled features align with specific semantic, syntactic, and pragmatic categories.<n>Our findings suggest that SAE-based decomposition offers a viable path toward transparent and steerable neural information retrieval.
  arXiv  Detail & Related papers  (2026-06-19T02:39:50Z)
• SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation [46.07708619404428]
  Seg is an end-to-end model that leverages a Sparse Autoencoder to forge an explicit, interpretable, and differentiable alignment pathway.<n>A "white-box" connection that is significantly more traceable than latent queries and more coherent than textual readouts.
  arXiv  Detail & Related papers  (2026-05-21T15:59:39Z)
• LUCID-SAE: Learning Unified Vision-Language Sparse Codes for Interpretable Concept Discovery [14.222802170483739]
  LUCID is a vision-language sparse autoencoder that learns a shared latent dictionary for image patch and text token representations.<n> LUCID yields interpretable shared features that support patch-level grounding, establish cross-modal neuron correspondence, and enhance robustness against the concept clustering problem.<n>Our analysis reveals that LUCID's shared features capture diverse semantic categories beyond objects, including actions, attributes, and abstract concepts.
  arXiv  Detail & Related papers  (2026-02-07T02:01:25Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Concepts [8.768503486874623]
  We propose a focused exploration framework that prioritizes curated concepts and their corresponding SAE features over attempts to visualize all available features simultaneously.<n>We present an interactive visualization system that combines topology-based visual encoding with dimensionality reduction to faithfully represent both local and global relationships among selected features.
  arXiv  Detail & Related papers  (2025-11-08T15:36:57Z)
• LAVA: Explainability for Unsupervised Latent Embeddings [0.0]
  Locality-Aware Variable Associations (LAVA) is designed to explain local embedding organization through its relationship with the input features.<n>Based on UMAP embeddings of MNIST and a single-cell kidney dataset, we show that LAVA captures relevant feature associations.
  arXiv  Detail & Related papers  (2025-09-25T13:38:17Z)
• Semantic Concentration for Self-Supervised Dense Representations Learning [103.10708947415092]
  Image-level self-supervised learning (SSL) has made significant progress, yet learning dense representations for patches remains challenging.<n>This work reveals that image-level SSL avoids over-dispersion by involving implicit semantic concentration.
  arXiv  Detail & Related papers  (2025-09-11T13:12:10Z)
• LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance [54.683384204063934]
  Large multi-modal models (LMMs) struggle with inaccurate segmentation and hallucinated comprehension.<n>We propose LIRA, a framework that capitalizes on the complementary relationship between visual comprehension and segmentation.<n>LIRA achieves state-of-the-art performance in both segmentation and comprehension tasks.
  arXiv  Detail & Related papers  (2025-07-08T07:46:26Z)
• SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning [38.507994878183474]
  We introduce Semantically contextualized VIsual Patches (SVIP) for Zero-shot learning (ZSL)<n>We propose a self-supervised patch selection mechanism that preemptively learns to identify semantic-unrelated patches in the input space.<n>SVIP achieves state-of-the-art performance results while providing more interpretable and semantically rich feature representations.
  arXiv  Detail & Related papers  (2025-03-13T10:59:51Z)
• Disentangling Dense Embeddings with Sparse Autoencoders [0.0]
  Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks. We present one of the first applications of SAEs to dense text embeddings from large language models. We show that the resulting sparse representations maintain semantic fidelity while offering interpretability.
  arXiv  Detail & Related papers  (2024-08-01T15:46:22Z)
• Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning [82.29761875805369]
  One of the ultimate goals of representation learning is to achieve compactness within a class and well-separability between classes. We propose a novel perspective to use pre-defined class anchors serving as feature centroid to unidirectionally guide feature learning. The proposed Semantic Anchor Regularization (SAR) can be used in a plug-and-play manner in the existing models.
  arXiv  Detail & Related papers  (2023-12-19T05:52:38Z)
• Closed-Form Factorization of Latent Semantics in GANs [65.42778970898534]
  A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. We propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
  arXiv  Detail & Related papers  (2020-07-13T18:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.