PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding

• URL: http://arxiv.org/abs/2602.01322v1
• Date: Sun, 01 Feb 2026 16:34:45 GMT
• Title: PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding
• Abstract summary: We introduce PolySAE, which extends the SAE decoder with higher-order terms to model feature interactions.<n>Across four language models and three SAE variants, PolySAE achieves an average improvement of approximately 8% in probing F1.
• Score: 16.10987386200554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse autoencoders (SAEs) have emerged as a promising method for interpreting neural network representations by decomposing activations into sparse combinations of dictionary atoms. However, SAEs assume that features combine additively through linear reconstruction, an assumption that cannot capture compositional structure: linear models cannot distinguish whether "Starbucks" arises from the composition of "star" and "coffee" features or merely their co-occurrence. This forces SAEs to allocate monolithic features for compound concepts rather than decomposing them into interpretable constituents. We introduce PolySAE, which extends the SAE decoder with higher-order terms to model feature interactions while preserving the linear encoder essential for interpretability. Through low-rank tensor factorization on a shared projection subspace, PolySAE captures pairwise and triple feature interactions with small parameter overhead (3% on GPT2). Across four language models and three SAE variants, PolySAE achieves an average improvement of approximately 8% in probing F1 while maintaining comparable reconstruction error, and produces 2-10$\times$ larger Wasserstein distances between class-conditional feature distributions. Critically, learned interaction weights exhibit negligible correlation with co-occurrence frequency ($r = 0.06$ vs. $r = 0.82$ for SAE feature covariance), suggesting that polynomial terms capture compositional structure, such as morphological binding and phrasal composition, largely independent of surface statistics.

Related papers

• Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling [55.36837305276515]
  We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling.<n>We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost.
  arXiv  Detail & Related papers  (2026-08-29T11:43:25Z)
• Understanding Synergistic Interactions among Pathology Foundation Models via Adaptive Fusion [49.550545038402184]
  Pathology foundation models (PFMs) provide strong tile-level representations via self-supervised pre-training on large-scale pathology images.<n>We present AdaFusion, a lightweight adaptive fusion framework that integrates complementary signals from multiple frozen PFMs.<n>AdaFusion consistently outperforms individual PFMs and other fusion baselines, while providing interpretable tissue visualisation.
  arXiv  Detail & Related papers  (2026-08-02T16:44:45Z)
• All you need is SAMPAT [4.978871870250063]
  Interpretability is crucial to gleaning insights while analyzing experimental data.<n>We present a three layer neural architecture, SAMPAT, that can provably learn a continuous, everywhere differentiable function.<n>Experiments on synthetic and benchmark datasets indicate that SAMPAT yields competitive performance with simpler representations.
  arXiv  Detail & Related papers  (2026-07-10T09:31:01Z)
• PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding [2.360330081206126]
  We introduce PairSAE, which summarizes pairwise tensors via an N-mode SVD into token-wise interaction roles.<n>We then use a sparse autoencoder to learn a shared set of token-level features that decode into both sequence and pair representations.<n>These results indicate that PairSAE links the latent space of foundation models for structural biology to interpretable structural concepts.
  arXiv  Detail & Related papers  (2026-06-25T18:01:44Z)
• The Representational Limit of Scalar Interactions: An Interventional Decomposition [9.850981754860754]
  We prove that signed pairwise interaction scores fundamentally conflate uniqueness (U), redundancy (R), and synergy (S)<n>We introduce Hi-Fi, a post-hoc, retraining-free predictability decomposition that estimates per-feature U/R/S profiles by interventional masked inference.
  arXiv  Detail & Related papers  (2026-06-17T15:21:28Z)
• Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models [51.56484100374058]
  We develop a framework for estimating how many near-orthogonal directions a model can support.<n>Applying this metric across dozens of open-source models reveals two classes: models with high $varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $varepsilon$ that maintain it.
  arXiv  Detail & Related papers  (2026-06-01T18:28:56Z)
• Feature Starvation as Geometric Instability in Sparse Autoencoders [0.764671395172401]
  We introduce adaptive elastic net SAEs, a fully differentiable architecture grounded in classical sparse regression.<n>AEN-SAEs combine an $ell$ structural term that enforces strong convexity and Lipschitz stability with adaptive $ell$ reweighting.<n>We show that AEN-SAEs yield a Lipschitzcontinuous sparse coding map and recover the global feature support under mild assumptions.
  arXiv  Detail & Related papers  (2026-05-06T18:11:14Z)
• RiboSphere: Learning Unified and Efficient Representations of RNA Structures [57.40815107640066]
  RNA backbones are highly flexible, non-canonical interactions are prevalent, and experimentally determined 3D structures are comparatively scarce.<n>We introduce emphRiboSphere, a framework that learns emphdiscrete geometric representations of RNA by combining vector quantization with flow matching.
  arXiv  Detail & Related papers  (2026-03-20T04:36:35Z)
• RaCo: Ranking and Covariance for Practical Learned Keypoints [51.38393049306958]
  RaCo is designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks.<n>RaCo operates without the need for covisible image pairs.
  arXiv  Detail & Related papers  (2026-02-17T17:39:52Z)
• Scalable and Interpretable Scientific Discovery via Sparse Variational Gaussian Process Kolmogorov-Arnold Networks (SVGP KAN) [0.0]
  Kolmogorov-Arnold Networks (KANs) offer a promising alternative to Multi-Layer Perceptron (MLP)<n>KANs lack probabilistic outputs, limiting their utility in applications requiring uncertainty quantification.<n>We introduce the Sparse Variational GP-KAN, an architecture that integrates sparse variational inference with the KAN topology.
  arXiv  Detail & Related papers  (2025-11-29T00:48:55Z)
• Pearl: A Foundation Model for Placing Every Atom in the Right Location [52.35027831422145]
  We introduce Pearl, a foundation model for protein-ligand cofolding at scale.<n>Pearl establishes a new state-of-the-art performance in protein-ligand cofolding.<n>Pearl surpasses AlphaFold 3 and other open source baselines on the public Runs N' Poses and PoseBusters benchmarks.
  arXiv  Detail & Related papers  (2025-10-28T17:36:51Z)
• Randomized based restricted kernel machine for hyperspectral image classification [0.0]
  Random vector functional link (RVFL) network has gained significant popularity in hyperspectral image (HSI) classification.<n> RVFL models face several limitations, particularly in handling non-linear relationships and complex data structures.<n>We propose a novel randomized based restricted kernel machine ($R2KM$) model that combines the strehyperngths of RVFL and restricted kernel machines.
  arXiv  Detail & Related papers  (2025-03-06T17:18:39Z)
• SAFR: Neuron Redistribution for Interpretability [7.756342860929851]
  Superposition refers to encoding representations of multiple features within a single neuron.<n>Despite promising performance, the model's interpretability has been diminished.<n>This paper presents a novel approach to enhance model interpretability by regularizing feature superposition.
  arXiv  Detail & Related papers  (2025-01-23T06:20:33Z)
• Incorporating Arbitrary Matrix Group Equivariance into KANs [69.30866522377694]
  Kolmogorov-Arnold Networks (KANs) have seen great success in scientific domains.<n>We propose Equivariant Kolmogorov-Arnold Networks (EKAN) to broaden their applicability to more fields.
  arXiv  Detail & Related papers  (2024-10-01T06:34:58Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)
• SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning [3.775565013663731]
  We propose a new method -- Sparser Random Feature Models via IMP (ShRIMP) -- to efficiently fit high-dimensional data with inherent low-dimensional structure. Our method can be viewed as a combined process to construct and find sparse lottery tickets for two-layer dense networks.
  arXiv  Detail & Related papers  (2021-12-07T21:32:28Z)
• Bilinear Classes: A Structural Framework for Provable Generalization in RL [119.42509700822484]
  Bilinear Classes is a new structural framework which permits generalization in reinforcement learning. The framework incorporates nearly all existing models in which a sample complexity is achievable. Our main result provides an RL algorithm which has sample complexity for Bilinear Classes.
  arXiv  Detail & Related papers  (2021-03-19T16:34:20Z)
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution [57.635467829558664]
  We introduce a structural regularization across convolutional kernels in a CNN. We show that CNNs now maintain performance with dramatic reduction in parameters and computations.
  arXiv  Detail & Related papers  (2020-09-04T20:41:47Z)
• Reduced Dilation-Erosion Perceptron for Binary Classification [1.3706331473063877]
  Dilation-erosion perceptron (DEP) is a neural network obtained by a convex combination of a dilation and an erosion. This paper introduces the reduced dilation-erosion (r-DEP) classifier.
  arXiv  Detail & Related papers  (2020-03-04T19:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.