Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?

• URL: http://arxiv.org/abs/2606.22994v1
• Date: Mon, 22 Jun 2026 08:12:34 GMT
• Title: Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?
• Abstract summary: We derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery.<n>Applying this protocol to current SAE approaches trained on visual data, we find that establishing good hierarchical structure remains challenging.<n>In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality.
• Score: 34.5057230491607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces more interpretable and manageable, recent approaches have begun imposing hierarchical structure, either explicitly or as an implicit effect of training constraints, yet rigorous comparison remains difficult. There are no agreed-upon requirements for what a meaningful feature hierarchy should satisfy, and evaluation has largely relied on qualitative illustrations with fragmented quantitative protocols. To address this, we derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery, drawing on semantic net and taxonomy research alongside recent SAE work, and use them to derive a concrete evaluation protocol. Applying this protocol to current SAE approaches trained on visual data, we find that while feature spaces generally provide a basis for sensible hierarchies, establishing good hierarchical structure remains challenging. In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality, pointing to a fundamental tension that future approaches will need to navigate.

Related papers

• A Unified Structured Query Understanding Framework for Industrial Semantic Search [9.913119853395205]
  Query understanding in large-scale industrial search systems is typically implemented as a cascade of disparate, task-specific components.<n>We propose and deploy a unified structured query understanding system that consolidates these heterogeneous functions into a single Small Language Model.<n>We validate this approach through extensive offline and online tests within LinkedIn's Job Search system.
  arXiv  Detail & Related papers  (2026-05-22T19:35:15Z)
• Online Continual Learning with Dynamic Label Hierarchies [72.33335166136045]
  We introduce a new problem setting, DHOCL, where severity evolve across granularities and each sample provides supervision at a single hierarchical level.<n>In this setting, we find two fundamental issues: (i) partial supervision under mixed granularities provides only point-wise signals over an evolving path-wise hierarchy, which constrains plasticity and undermines cross-level semantic consistency.<n>We propose HALO, which adaptively combines complementary classification heads, regularized by organized learnable hierarchical prototypes, enabling rapid adaptation, hierarchical consistency, and structured knowledge consolidation as the taxonomy evolves.
  arXiv  Detail & Related papers  (2026-05-12T08:20:23Z)
• SC-Taxo: Hierarchical Taxonomy Generation under Semantic Consistency Constraints using Large Language Models [24.295783768613344]
  A high-quality scientific taxonomy offers a structured and hierarchical representation of a research field.<n>Existing taxonomy generation approaches often suffer from structural inconsistencies and semantic misalignment across hierarchical levels.
  arXiv  Detail & Related papers  (2026-05-01T12:42:44Z)
• Beyond the Academic Monoculture: A Unified Framework and Industrial Perspective for Attributed Graph Clustering [75.50670592447219]
  Attributed Graph Clustering (AGC) is a fundamental unsupervised task that partitions nodes into cohesive groups by jointly modeling structural topology and node attributes.<n>This survey provides a comprehensive, industrially grounded review of AGC from three complementary perspectives.
  arXiv  Detail & Related papers  (2026-03-21T14:15:34Z)
• Digging Deeper: Learning Multi-Level Concept Hierarchies [23.16720677779406]
  Concept-based models typically rely on exhaustive annotations to explain predictions with human-understandable concepts.<n>We introduce Multi-Level Concept Splitting (MLCS), which discovers multi-level concept hierarchies from only top-level supervision.<n>We also present Deep-HiCEMs, an architecture that represents these discovered hierarchies and enables interventions at multiple levels of abstraction.
  arXiv  Detail & Related papers  (2026-03-10T12:48:20Z)
• Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining [58.69016084278948]
  We consider a hierarchical context-free grammar introduced by arXiv:2307.02129 and conjectured to separate deep and shallow networks.<n>We prove that, under mild conditions, a deep convolutional network can be efficiently trained to learn this function class.
  arXiv  Detail & Related papers  (2026-01-27T16:19:54Z)
• A Semantics-Aware Hierarchical Self-Supervised Approach to Classification of Remote Sensing Images [12.282079123411947]
  We present a novel Semantics-Aware Hierarchical Consensus (SAHC) method for learning hierarchical features and relationships.<n>The SAHC method is evaluated on three benchmark datasets with different degrees of hierarchical complexity.<n> Experimental results show both the effectiveness of the proposed approach in guiding network learning and the robustness of the hierarchical consensus for remote sensing image classification tasks.
  arXiv  Detail & Related papers  (2025-10-06T15:30:39Z)
• Predicate Hierarchies Improve Few-Shot State Classification [12.683069420087962]
  PHIER is an object-centric scene encoder, self-supervised losses that infer semantic relations between predicates, and a hyperbolic distance metric that captures hierarchical structure.<n>We evaluate PHIER in the CALVIN and BEHAVIOR robotic environments and show that PHIER significantly outperforms existing methods in few-shot, out-of-distribution state classification.
  arXiv  Detail & Related papers  (2025-02-18T03:08:37Z)
• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation.<n>Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy.<n>Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality [55.88910947643436]
  Self-supervised pre-training is essential for handling vast quantities of unlabeled data in practice. HiDe-Prompt is an innovative approach that explicitly optimize the hierarchical components with an ensemble of task-specific prompts and statistics. Our experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning.
  arXiv  Detail & Related papers  (2023-10-11T06:51:46Z)
• Provable Hierarchy-Based Meta-Reinforcement Learning [50.17896588738377]
  We analyze HRL in the meta-RL setting, where learner learns latent hierarchical structure during meta-training for use in a downstream task. We provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
  arXiv  Detail & Related papers  (2021-10-18T17:56:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.