From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders

• URL: http://arxiv.org/abs/2602.11881v1
• Date: Thu, 12 Feb 2026 12:30:23 GMT
• Title: From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders
• Abstract summary: Hierarchical Sparse Autoencoder (HSAE) learns a series of SAEs and the parent-child relationships between their features.<n>Our work provides a powerful, scalable tool for discovering and analyzing the multi-scale conceptual structures embedded in language representations.
• Score: 12.294297654425334
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sparse autoencoders (SAEs) have proven effective for extracting monosemantic features from large language models (LLMs), yet these features are typically identified in isolation. However, broad evidence suggests that LLMs capture the intrinsic structure of natural language, where the phenomenon of "feature splitting" in particular indicates that such structure is hierarchical. To capture this, we propose the Hierarchical Sparse Autoencoder (HSAE), which jointly learns a series of SAEs and the parent-child relationships between their features. HSAE strengthens the alignment between parent and child features through two novel mechanisms: a structural constraint loss and a random feature perturbation mechanism. Extensive experiments across various LLMs and layers demonstrate that HSAE consistently recovers semantically meaningful hierarchies, supported by both qualitative case studies and rigorous quantitative metrics. At the same time, HSAE preserves the reconstruction fidelity and interpretability of standard SAEs across different dictionary sizes. Our work provides a powerful, scalable tool for discovering and analyzing the multi-scale conceptual structures embedded in LLM representations.

Related papers

• Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models [47.868429337792314]
  Large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency.<n>We propose Hierarchical Representation Regularization ($HiR2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs.<n>$HiR2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods.
  arXiv  Detail & Related papers  (2026-07-03T03:16:41Z)
• Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs [12.597421665142646]
  We introduce cascaded autoencoders (CSAEs) for learning hierarchical visual concepts in MLLMs.<n> experiments across Qwen3-VL, Gemma-3, and LLaVA on multiple visual datasets show that CSAEs improve interpretability in terms of hierarchical concept coherence.<n>Results on concept steering further demonstrate that the learned concept groups support effective group-level interventions in MLLM outputs.
  arXiv  Detail & Related papers  (2026-06-15T04:10:40Z)
• Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoders [15.761881465116124]
  We introduce a novel reconstruction condition that enforces a deeper functional link between hierarchical levels.<n>By combining both activation and reconstruction constraints, we propose the Tree SAE, a model designed to learn hierarchical structures directly from within the feature set.<n>Our results demonstrate that Tree SAEs significantly surpass the existing SAEs at learning hierarchical pairs while maintaining competitive performance to the state-of-the-art on several key benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T15:57:37Z)
• SERE: Structural Example Retrieval for Enhancing LLMs in Event Causality Identification [50.174050426976514]
  Event Causality Identification (ECI) requires models to determine whether a given pair of events in a context exhibits a causal relationship.<n>Large Language Models (LLMs) have demonstrated strong performance across various NLP tasks, but their effectiveness in ECI remains limited due to biases in causal reasoning.<n>We propose SERE, a structural example retrieval framework that leverages LLMs' few-shot learning capabilities.
  arXiv  Detail & Related papers  (2026-05-05T12:50:19Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• Structure-aware Prompt Adaptation from Seen to Unseen for Open-Vocabulary Compositional Zero-Shot Learning [86.58227205147546]
  The goal of Compositional Zero-Shot Learning (OV-CZSL) is to recognize iteration-object compositions in the open-vocabulary setting.<n>We propose Structure-aware Prompt Adaptation (SPA) method, which enables models to generalize from seen to unseen attributes and objects.
  arXiv  Detail & Related papers  (2026-03-04T07:54:28Z)
• StructLens: A Structural Lens for Language Models via Maximum Spanning Trees [52.040177523973334]
  StructLens is an analytical framework designed to reveal how internal structures relate holistically.<n>Our findings demonstrate that StructLens yields an inter-layer similarity pattern that is distinctively different from conventional cosine similarity.
  arXiv  Detail & Related papers  (2026-02-10T11:30:32Z)
• Table-BiEval: A Self-Supervised, Dual-Track Framework for Decoupling Structure and Content in LLM Evaluation [11.450834626205676]
  Table-BiEval is a novel approach based on a human-free, self-supervised evaluation framework.<n>It calculates Content Semantic Accuracy and Normalized Tree Edit Distance to decouple structure from content.<n>Results reveal substantial variability, highlighting that mid-sized models can surprisingly outperform larger counterparts in structural efficiency.
  arXiv  Detail & Related papers  (2026-01-09T07:38:27Z)
• Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders [0.0]
  We present a novel SAE-based architecture tailored for text classification.<n>We benchmark this architecture against established methods such as ConceptShap, Independent Component Analysis, and other SAE-based concept extraction techniques.<n>Our empirical results show that our architecture improves both the causality and interpretability of the extracted features.
  arXiv  Detail & Related papers  (2025-06-30T15:18:50Z)
• NEXT: Multi-Grained Mixture of Experts via Text-Modulation for Multi-Modal Object Re-Identification [17.10113184019939]
  Multi-modal object Re-Identification (ReID) aims to obtain accurate identity features across heterogeneous modalities.<n>In this paper, we propose a reliable caption generation pipeline based on attribute confidence.<n>We also propose a novel ReID framework, named NEXT, to model diverse identity patterns.
  arXiv  Detail & Related papers  (2025-05-26T13:52:28Z)
• Interpreting CLIP with Hierarchical Sparse Autoencoders [8.692675181549117]
  Matryoshka SAE (MSAE) learns hierarchical representations at multiple granularities simultaneously.<n>MSAE establishes a new state-of-the-art frontier between reconstruction quality and sparsity for CLIP.
  arXiv  Detail & Related papers  (2025-02-27T22:39:13Z)
• The Complexity of Learning Sparse Superposed Features with Feedback [2.4140387101794283]
  We investigate whether the underlying learned features of a model can be efficiently retrieved through feedback from an agent.<n>We analyze the feedback complexity associated with learning a feature matrix in sparse settings.<n>Our results establish tight bounds when the agent is permitted to construct activations and demonstrate strong upper bounds in sparse scenarios.
  arXiv  Detail & Related papers  (2025-02-08T01:54:23Z)
• GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning [51.677086019209554]
  We propose a Generalized Structural Sparse to capture powerful relationships across modalities for pair-wise similarity learning. The distance metric delicately encapsulates two formats of diagonal and block-diagonal terms. Experiments on cross-modal and two extra uni-modal retrieval tasks have validated its superiority and flexibility.
  arXiv  Detail & Related papers  (2024-10-20T03:45:50Z)
• Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings [60.698130703909804]
  Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset. We propose SQ-Transformer that explicitly encourages systematicity in the embeddings and attention layers. We show that SQ-Transformer achieves stronger compositional generalization than the vanilla Transformer on multiple low-complexity semantic parsing and machine translation datasets.
  arXiv  Detail & Related papers  (2024-02-09T15:53:15Z)
• LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model [96.889634747943]
  Universally modeling all typical information extraction tasks (UIE) with one generative language model (GLM) has revealed great potential. We propose a novel structure-aware GLM, fully unleashing the power of syntactic knowledge for UIE. Over 12 IE benchmarks across 7 tasks our system shows significant improvements over the baseline UIE system.
  arXiv  Detail & Related papers  (2023-04-13T04:01:14Z)
• Provable Hierarchy-Based Meta-Reinforcement Learning [50.17896588738377]
  We analyze HRL in the meta-RL setting, where learner learns latent hierarchical structure during meta-training for use in a downstream task. We provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
  arXiv  Detail & Related papers  (2021-10-18T17:56:02Z)
• Structure-Aware Feature Generation for Zero-Shot Learning [108.76968151682621]
  We introduce a novel structure-aware feature generation scheme, termed as SA-GAN, to account for the topological structure in learning both the latent space and the generative networks. Our method significantly enhances the generalization capability on unseen-classes and consequently improve the classification performance.
  arXiv  Detail & Related papers  (2021-08-16T11:52:08Z)
• Tree-structured Attention with Hierarchical Accumulation [103.47584968330325]
  "Hierarchical Accumulation" encodes parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task.
  arXiv  Detail & Related papers  (2020-02-19T08:17:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.