Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features

• URL: http://arxiv.org/abs/2604.23829v2
• Date: Tue, 28 Apr 2026 16:11:54 GMT
• Title: Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features
• Abstract summary: We build a strict domain-specific concept universe from a large SAE inventory.<n>We then build two aligned graph views on the filtered set.<n>We label these graph views into readable knowledge graphs rather than unlabeled layouts.
• Score: 2.61135281451375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse autoencoders (SAEs) extract millions of interpretable features from a language model, but flat feature inventories aren't very useful on their own. Domain concepts get mixed with generic and weakly grounded features, while related ideas are scattered across many units, and there's no way to understand relationships between features. We address this by first constructing a strict domain-specific concept universe from a large SAE inventory using contrastive activations and a multi-stage filtering process. Next, we build two aligned graph views on the filtered set: a co-occurrence graph for corpus-level conceptual structure, organized at multiple levels of granularity, and a transcoder-based mechanism graph that links source-layer and target-layer features through sparse latent pathways. Automated edge labeling then turns these graph views into readable knowledge graphs rather than unlabeled layouts. In a case study on a biology textbook, these graphs recover coherent chapter and subchapter-level structure, reveal concepts that bridge neighboring topics, and transform messy sentence-level activity containing thousands of features into compact, readable views that illustrate the model's local activity. Taken together, this reframes a flat SAE inventory as an internal knowledge graph that converts feature-level interpretability into a global map of model knowledge and enables audits of reasoning faithfulness.

Related papers

• From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features [0.0]
  We introduce a graph-structured representation in which each SAE feature is modelled as a token co-occurrence graph.<n>A custom WL-style, frequency-binned graph kernel then provides a similarity measure over this structural space.<n>A token-histogram baseline characterises higher overall purity, so the contribution of the graph view is complementary rather than dominant.
  arXiv  Detail & Related papers  (2026-05-07T16:15:16Z)
• <SOG_k>: One LLM Token for Explicit Graph Structural Understanding [57.017902343605364]
  We propose to incorporate one special token SOG_k> to fully represent the Structure Of Graph within a unified token space.<n>SOG_k> empowers LLMs to understand, generate, and reason in a concise and accurate manner.
  arXiv  Detail & Related papers  (2026-02-02T07:55:09Z)
• Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation [79.75818239774952]
  Large language models (LLMs) have demonstrated remarkable capabilities, but still struggle with issues like hallucinations and outdated information.<n>Retrieval-augmented generation (RAG) addresses these issues by grounding LLM outputs in external knowledge with an Information Retrieval (IR) system.<n>We propose Align-GRAG, a novel reasoning-guided dual alignment framework in post-retrieval phrase.
  arXiv  Detail & Related papers  (2025-05-22T05:15:27Z)
• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Individual and Structural Graph Information Bottlenecks for Out-of-Distribution Generalization [21.227825123510293]
  We propose Individual Graph Information Bottleneck (I-GIB) and Structural Graph Information Bottleneck (S-GIB) I-GIB discards irrelevant information by minimizing the mutual information between the input graph and its embeddings. S-GIB simultaneously discards spurious features and learn invariant features from a high-order perspective.
  arXiv  Detail & Related papers  (2023-06-28T03:52:41Z)
• Multi-task Self-distillation for Graph-based Semi-Supervised Learning [6.277952154365413]
  We propose a multi-task self-distillation framework that injects self-supervised learning and self-distillation into graph convolutional networks. First, we formulate a self-supervision pipeline based on pre-text tasks to capture different levels of similarities in graphs. Second, self-distillation uses soft labels of the model itself as additional supervision.
  arXiv  Detail & Related papers  (2021-12-02T12:43:41Z)
• Multi-Level Graph Contrastive Learning [38.022118893733804]
  We propose a Multi-Level Graph Contrastive Learning (MLGCL) framework for learning robust representation of graph data by contrasting space views of graphs. The original graph is first-order approximation structure and contains uncertainty or error, while the $k$NN graph generated by encoding features preserves high-order proximity. Extensive experiments indicate MLGCL achieves promising results compared with the existing state-of-the-art graph representation learning methods on seven datasets.
  arXiv  Detail & Related papers  (2021-07-06T14:24:43Z)
• Self-supervised Graph-level Representation Learning with Local and Global Structure [71.45196938842608]
  We propose a unified framework called Local-instance and Global-semantic Learning (GraphLoG) for self-supervised whole-graph representation learning. Besides preserving the local similarities, GraphLoG introduces the hierarchical prototypes to capture the global semantic clusters. An efficient online expectation-maximization (EM) algorithm is further developed for learning the model.
  arXiv  Detail & Related papers  (2021-06-08T05:25:38Z)
• Graphonomy: Universal Image Parsing via Graph Reasoning and Transfer [140.72439827136085]
  We propose a graph reasoning and transfer learning framework named "Graphonomy" It incorporates human knowledge and label taxonomy into the intermediate graph representation learning beyond local convolutions. It learns the global and structured semantic coherency in multiple domains via semantic-aware graph reasoning and transfer.
  arXiv  Detail & Related papers  (2021-01-26T08:19:03Z)
• HOSE-Net: Higher Order Structure Embedded Network for Scene Graph Generation [20.148175528691905]
  This paper presents a novel structure-aware embedding-to-classifier(SEC) module to incorporate both local and global structural information of relationships into the output space. We also propose a hierarchical semantic aggregation(HSA) module to reduce the number of subspaces by introducing higher order structural information. The proposed HOSE-Net achieves the state-of-the-art performance on two popular benchmarks of Visual Genome and VRD.
  arXiv  Detail & Related papers  (2020-08-12T07:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.