From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features

• URL: http://arxiv.org/abs/2605.06494v1
• Date: Thu, 07 May 2026 16:15:16 GMT
• Title: From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features
• Abstract summary: We introduce a graph-structured representation in which each SAE feature is modelled as a token co-occurrence graph.<n>A custom WL-style, frequency-binned graph kernel then provides a similarity measure over this structural space.<n>A token-histogram baseline characterises higher overall purity, so the contribution of the graph view is complementary rather than dominant.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse autoencoders (SAEs) have become central to mechanistic interpretability, decomposing transformer activations into monosemantic features. Yet existing analyses characterise features almost exclusively through top-activating token lists or decoder weight vectors, leaving the higher-order co-occurrence structure shared across features largely unexamined. We introduce a graph-structured representation in which each SAE feature is modelled as a token co-occurrence graph: nodes are the tokens most frequent near strong activations, and edges connect pairs that co-occur within local context windows. A custom WL-style, frequency-binned graph kernel then provides a similarity measure over this structural space. Applied as a proof of concept to features from a large SAE trained on GPT-2 Small and probed with a synthetic mixed-domain corpus, our clustering recovers heuristic motif families (punctuation-heavy patterns, language and script clusters, and code-like templates) that are not recovered by clustering on decoder cosine similarity. A token-histogram baseline achieves higher overall purity, so the contribution of the graph view is complementary rather than dominant: it surfaces structural relationships that token-frequency and decoder-weight views alone do not capture. Cluster assignments are stable across graph-construction hyperparameters and random seeds.

Related papers

• Quantum-Inspired Contextual Learning for Sparse-Ring Fraud Detection in Dynamic Transaction Graphs [0.0]
  Coordinated fraud may not be visible from individual transactions alone, but may emerge as a multi-period relational pattern.<n>We focus on sparse-ring fraud, a stylized pattern in which a completed directed cycle is distributed across several days.<n>We study this problem using a synthetic transaction simulator with completed sparse-ring injections and broken-ring decoys.
  arXiv  Detail & Related papers  (2026-06-22T05:38:02Z)
• MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging [41.25982557850732]
  We introduce MergeTok, a unified tokenizer that jointly optimize continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture.<n> MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets.
  arXiv  Detail & Related papers  (2026-05-29T06:38:10Z)
• SCGNN: Semantic Consistency enhanced Graph Neural Network Guided by Granular-ball Computing [21.531111082932284]
  Capturing semantic consistency among nodes is crucial for effective graph representation learning.<n>We propose the Semantic Consistency enhanced Graph Neural Network (SCGNN), a novel plug-and-play framework.<n>SCGNN models group-level semantic structure by adaptively partitioning nodes into granular balls.
  arXiv  Detail & Related papers  (2026-05-04T14:09:56Z)
• Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features [2.61135281451375]
  We build a strict domain-specific concept universe from a large SAE inventory.<n>We then build two aligned graph views on the filtered set.<n>We label these graph views into readable knowledge graphs rather than unlabeled layouts.
  arXiv  Detail & Related papers  (2026-04-26T18:17:56Z)
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging [65.07273789940116]
  This paper introduces a hierarchical architecture that jointly optimize a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks.<n> MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation.
  arXiv  Detail & Related papers  (2025-11-17T19:27:41Z)
• Symbolic Graph Intelligence: Hypervector Message Passing for Learning Graph-Level Patterns with Tsetlin Machines [0.0]
  We propose a multilayered symbolic framework for general graph classification.<n>Each graph is encoded through structured message passing, where node, edge, and attribute information are bound and bundled into a symbolic hypervector.<n>We validate our method on TUDataset benchmarks, demonstrating competitive accuracy with strong symbolic transparency compared to neural graph models.
  arXiv  Detail & Related papers  (2025-07-22T12:47:56Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Graph Spectral Embedding using the Geodesic Betweeness Centrality [76.27138343125985]
  We introduce the Graph Sylvester Embedding (GSE), an unsupervised graph representation of local similarity, connectivity, and global structure. GSE uses the solution of the Sylvester equation to capture both network structure and neighborhood proximity in a single representation.
  arXiv  Detail & Related papers  (2022-05-07T04:11:23Z)
• Spectral Embedding of Graph Networks [76.27138343125985]
  We introduce an unsupervised graph embedding that trades off local node similarity and connectivity, and global structure. The embedding is based on a generalized graph Laplacian, whose eigenvectors compactly capture both network structure and neighborhood proximity in a single representation.
  arXiv  Detail & Related papers  (2020-09-30T04:59:10Z)
• Graph Neural Networks with Composite Kernels [60.81504431653264]
  We re-interpret node aggregation from the perspective of kernel weighting. We present a framework to consider feature similarity in an aggregation scheme. We propose feature aggregation as the composition of the original neighbor-based kernel and a learnable kernel to encode feature similarities in a feature space.
  arXiv  Detail & Related papers  (2020-05-16T04:44:29Z)
• Embedding Graph Auto-Encoder for Graph Clustering [90.8576971748142]
  Graph auto-encoder (GAE) models are based on semi-supervised graph convolution networks (GCN) We design a specific GAE-based model for graph clustering to be consistent with the theory, namely Embedding Graph Auto-Encoder (EGAE) EGAE consists of one encoder and dual decoders.
  arXiv  Detail & Related papers  (2020-02-20T09:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.