Finding Manifolds With Bilinear Autoencoders

• URL: http://arxiv.org/abs/2510.16820v1
• Date: Sun, 19 Oct 2025 13:17:14 GMT
• Title: Finding Manifolds With Bilinear Autoencoders
• Abstract summary: Polynomials serve as primitives that can be analysed without reference to input.<n>We discuss improvements that induce importance ordering, clustering, and activation sparsity.<n>This is an initial step toward analysable latents through their properties.
• Score: 0.8594140167290097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse autoencoders are a standard tool for uncovering interpretable latent representations in neural networks. Yet, their interpretation depends on the inputs, making their isolated study incomplete. Polynomials offer a solution; they serve as algebraic primitives that can be analysed without reference to input and can describe structures ranging from linear concepts to complicated manifolds. This work uses bilinear autoencoders to efficiently decompose representations into quadratic polynomials. We discuss improvements that induce importance ordering, clustering, and activation sparsity. This is an initial step toward nonlinear yet analysable latents through their algebraic properties.

Related papers

• Flexible Online Representation Learning Based on Similarity Matching [2.461513633983538]
  Sparse high-dimensional representations are conducive to uncovering nontrivial structures in unsupervised exploration of data.<n>We propose a versatile online biologically plausible learning algorithm capable of learning sparse shift-invariant representations.
  arXiv  Detail & Related papers  (2026-06-01T01:46:27Z)
• Bilinear autoencoders find interpretable manifolds [0.6999740786886536]
  We implement quadratic latents with bilinear autoencoders, which decompose activations into low-rank quadratic forms, compose linearly in weight space, and admit input-independent geometric analysis.<n>Our experiments and visualisations show that multi-dimensional geometries are highly prevalent and that composite latents capture them well.<n> Practically, these models serve as an unsupervised tool for manifold discovery, which we demonstrate through an interactive online visualizer for Qwen 3.5.
  arXiv  Detail & Related papers  (2026-05-09T11:16:34Z)
• Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints [5.104181562775778]
  We show that Linear probes and sparse autoencoders consistently recover meaningful structure from transformer representations.<n>We formalize this as the emphInvariant Subspace Necessity theorem and derive the emphSelf-Reference Property: tokens directly provide the geometric direction for their associated features.
  arXiv  Detail & Related papers  (2026-02-10T13:42:55Z)
• The Origins of Representation Manifolds in Large Language Models [52.68554895844062]
  We show that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths.<n>The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.
  arXiv  Detail & Related papers  (2025-05-23T13:31:22Z)
• Unveiling and Steering Connectome Organization with Interpretable Latent Variables [11.702970031377307]
  We propose a framework that combines subgraph extraction from the Drosophila connectome, FlyWire, with a generative model to derive interpretable low-dimensional representations of neural circuitry.<n>This research offers a novel tool for understanding brain architecture and a potential avenue for designing bio-inspired artificial neural networks.
  arXiv  Detail & Related papers  (2025-05-19T11:54:40Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [76.15163242945813]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence.<n>We introduce a novel generative model that generates tokens on the basis of human-interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• Bilinear Convolution Decomposition for Causal RL Interpretability [0.0]
  Efforts to interpret reinforcement learning (RL) models often rely on high-level techniques such as attribution or probing.<n>This work proposes replacing nonlinearities in convolutional neural networks (ConvNets) with bilinear variants, to produce a class of models for which these limitations can be addressed.<n>We show bilinear model variants perform comparably in model-free reinforcement learning settings, and give a side by side comparison on ProcGen environments.
  arXiv  Detail & Related papers  (2024-12-01T19:32:04Z)
• Learning Linear Attention in Polynomial Time [127.14106124669645]
  We provide the first results on learnability of single-layer Transformers with linear attention.<n>We show that linear attention may be viewed as a linear predictor in a suitably defined RKHS.<n>We show how to efficiently identify training datasets for which every empirical riskr is equivalent to the linear Transformer.
  arXiv  Detail & Related papers  (2024-10-14T02:41:01Z)
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models [55.19497659895122]
  We introduce methods for discovering and applying sparse feature circuits.<n>These are causally implicatedworks of human-interpretable features for explaining language model behaviors.
  arXiv  Detail & Related papers  (2024-03-28T17:56:07Z)
• A technical note on bilinear layers for interpretability [0.0]
  Bilinear layers are a type of layer that are mathematically much easier to analyze. We can integrate this expression for bilinear layers into a mathematical framework for transformer circuits.
  arXiv  Detail & Related papers  (2023-05-05T11:56:26Z)
• Semi-Supervised Manifold Learning with Complexity Decoupled Chart Autoencoders [45.29194877564103]
  This work introduces a chart autoencoder with an asymmetric encoding-decoding process that can incorporate additional semi-supervised information such as class labels. We discuss the approximation power of such networks and derive a bound that essentially depends on the intrinsic dimension of the data manifold rather than the dimension of ambient space.
  arXiv  Detail & Related papers  (2022-08-22T19:58:03Z)
• Decoupling multivariate functions using a nonparametric filtered tensor decomposition [0.29360071145551075]
  Decoupling techniques aim at providing an alternative representation of the nonlinearity. The so-called decoupled form is often a more efficient parameterisation of the relationship while being highly structured, favouring interpretability. In this work two new algorithms, based on filtered tensor decompositions of first order derivative information are introduced.
  arXiv  Detail & Related papers  (2022-05-23T09:34:17Z)
• Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems [107.3868459697569]
  We introduce an eigendecomposition-free approach to training a deep network. We show that our approach is much more robust than explicit differentiation of the eigendecomposition. Our method has better convergence properties and yields state-of-the-art results.
  arXiv  Detail & Related papers  (2020-04-15T04:29:34Z)
• Learning Bijective Feature Maps for Linear ICA [73.85904548374575]
  We show that existing probabilistic deep generative models (DGMs) which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. We create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.
  arXiv  Detail & Related papers  (2020-02-18T17:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.