Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoders

• URL: http://arxiv.org/abs/2605.07922v2
• Date: Mon, 11 May 2026 02:18:14 GMT
• Title: Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoders
• Abstract summary: We introduce a novel reconstruction condition that enforces a deeper functional link between hierarchical levels.<n>By combining both activation and reconstruction constraints, we propose the Tree SAE, a model designed to learn hierarchical structures directly from within the feature set.<n>Our results demonstrate that Tree SAEs significantly surpass the existing SAEs at learning hierarchical pairs while maintaining competitive performance to the state-of-the-art on several key benchmarks.
• Score: 15.761881465116124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning hierarchical features in Sparse Autoencoders (SAEs) is essential for capturing the structured nature of real-world data and mitigating issues like feature absorption or splitting. Existing works attempt to identify hierarchical relationships within independent feature sets by relying on activation coverage, the assumption that child feature should only activate when its parent feature activates. However, we demonstrate that this condition alone is insufficient; that is, it often produces false positives where parent and child concepts are semantically unrelated. To address this, we introduce a novel reconstruction condition that enforces a deeper functional link between hierarchical levels. By combining both activation and reconstruction constraints, we propose the Tree SAE, a model designed to learn hierarchical structures directly from within the feature set. Our results demonstrate that Tree SAEs significantly surpass the existing SAEs at learning hierarchical pairs while maintaining competitive performance to the state-of-the-art on several key benchmarks. Finally, we demonstrate the practical utility of our Tree SAE in mapping the geometry of child feature subspaces and uncovering the complex hierarchical concept structures encoded within large language models.

Related papers

• Do Sparse Autoencoders Learn Meaningful Concept Hierarchies? [34.5057230491607]
  We derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery.<n>Applying this protocol to current SAE approaches trained on visual data, we find that establishing good hierarchical structure remains challenging.<n>In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality.
  arXiv  Detail & Related papers  (2026-06-22T08:12:34Z)
• Integrating SAINT with Tree-Based Models: A Case Study in Employee Attrition Prediction [0.0]
  This study explores a hybrid approach by integrating SAINT-generated embeddings with tree-based models to enhance employee attrition prediction.<n>We evaluate the performance, generalizability, and interpretability of standalone models (SAINT, XGBoost, LightGBM) and hybrid models that combine SAINT embeddings with tree-based classifiers.
  arXiv  Detail & Related papers  (2026-04-11T19:51:01Z)
• ASTRA: Adaptive Semantic Tree Reasoning Architecture for Complex Table Question Answering [55.55968342644846]
  Table serialization remains a critical bottleneck for Large Language Models (LLMs) in complex table question answering.<n>Existing serialization methods fail to capture explicit hierarchies and lack schema flexibility.<n>We propose ASTRA (Adaptive Semantic Tree Reasoning Architecture) including two main modules, AdaSTR and DuTR.<n> Experiments on complex table benchmarks demonstrate that our method achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-04-10T06:09:41Z)
• Learning Order Forest for Qualitative-Attribute Data Clustering [52.612779710298526]
  This paper discovers a tree-like distance structure to flexibly represent the local order relationship among intra-attribute qualitative values.<n>A joint learning mechanism is proposed to iteratively obtain more appropriate tree structures and clusters.<n>Experiments demonstrate that the joint learning adapts the forest to the clustering task to yield accurate results.
  arXiv  Detail & Related papers  (2026-03-03T07:49:50Z)
• Owen-based Semantics and Hierarchy-Aware Explanation (O-Shap) [6.096130289291087]
  In vision tasks, features often exhibit strong spatial and semantic dependencies.<n>Modern SHAP implementations now include the Owen value, a hierarchical generalization of the Shapley value that supports group attributions.<n>We show that the Owen value preserves the foundations of Shapley values, but its effectiveness critically depends on how feature groups are defined.<n>We propose a new segmentation approach that satisfies the $T$-property to ensure semantic alignment across hierarchy levels.
  arXiv  Detail & Related papers  (2026-02-19T06:06:12Z)
• From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders [12.294297654425334]
  Hierarchical Sparse Autoencoder (HSAE) learns a series of SAEs and the parent-child relationships between their features.<n>Our work provides a powerful, scalable tool for discovering and analyzing the multi-scale conceptual structures embedded in language representations.
  arXiv  Detail & Related papers  (2026-02-12T12:30:23Z)
• Hierarchical Quantized Diffusion Based Tree Generation Method for Hierarchical Representation and Lineage Analysis [49.00783841494125]
  HDTree captures tree relationships within a hierarchical latent space using a unified hierarchical codebook and quantized diffusion processes.<n> HDTree's effectiveness is demonstrated through comparisons on both general-purpose and single-cell datasets.<n>These contributions provide a new tool for hierarchical lineage analysis, enabling more accurate and efficient modeling of cellular differentiation paths.
  arXiv  Detail & Related papers  (2025-06-29T15:19:13Z)
• ReTreever: Tree-based Coarse-to-Fine Representations for Retrieval [64.44265315244579]
  We propose a tree-based method for organizing and representing reference documents at various granular levels.<n>Our method, called ReTreever, jointly learns a routing function per internal node of a binary tree such that query and reference documents are assigned to similar tree branches.<n>Our evaluations show that ReTreever generally preserves full representation accuracy.
  arXiv  Detail & Related papers  (2025-02-11T21:35:13Z)
• Integrating Hierarchical Semantic into Iterative Generation Model for Entailment Tree Explanation [7.5496857647335585]
  We propose an architecture of integrating the Hierarchical Semantics of sentences under the framework of Controller-Generator (HiSCG) to explain answers. The proposed method achieves comparable performance on all three settings of the EntailmentBank dataset.
  arXiv  Detail & Related papers  (2024-09-26T11:46:58Z)
• GrootVL: Tree Topology is All You Need in State Space Model [66.36757400689281]
  GrootVL is a versatile multimodal framework that can be applied to both visual and textual tasks. Our method significantly outperforms existing structured state space models on image classification, object detection and segmentation. By fine-tuning large language models, our approach achieves consistent improvements in multiple textual tasks at minor training cost.
  arXiv  Detail & Related papers  (2024-06-04T15:09:29Z)
• Hierarchical clustering with dot products recovers hidden tree structure [53.68551192799585]
  In this paper we offer a new perspective on the well established agglomerative clustering algorithm, focusing on recovery of hierarchical structure. We recommend a simple variant of the standard algorithm, in which clusters are merged by maximum average dot product and not, for example, by minimum distance or within-cluster variance. We demonstrate that the tree output by this algorithm provides a bona fide estimate of generative hierarchical structure in data, under a generic probabilistic graphical model.
  arXiv  Detail & Related papers  (2023-05-24T11:05:12Z)
• Hierarchical Dependency Constrained Tree Augmented Naive Bayes Classifiers for Hierarchical Feature Spaces [0.0]
  We propose two novel Hierarchical dependency-based Tree Augmented Naive Bayes algorithms, i.e. Hie-TAN and Hie-TAN-Lite. Hie-TAN successfully obtained better predictive performance than several other hierarchical dependency constrained classification algorithms.
  arXiv  Detail & Related papers  (2022-02-08T19:16:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.