On interpretability and proper latent decomposition of autoencoders

• URL: http://arxiv.org/abs/2211.08345v1
• Date: Tue, 15 Nov 2022 17:55:35 GMT
• Title: On interpretability and proper latent decomposition of autoencoders
• Abstract summary: This paper theoretically interpret the transformations of an autoencoder. We mathematically derive the metric tensor, which provides a mathematical description of the manifold. We propose a method -- proper latent decomposition (PLD) -- that generalizes proper decomposition of turbulent flows on the autoencoder latent space.
• Score: 6.85316573653194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The dynamics of a turbulent flow tend to occupy only a portion of the phase space at a statistically stationary regime. From a dynamical systems point of view, this portion is the attractor. The knowledge of the turbulent attractor is useful for two purposes, at least: (i) We can gain physical insight into turbulence (what is the shape and geometry of the attractor?), and (ii) it provides the minimal number of degrees of freedom to accurately describe the turbulent dynamics. Autoencoders enable the computation of an optimal latent space, which is a low-order representation of the dynamics. If properly trained and correctly designed, autoencoders can learn an approximation of the turbulent attractor, as shown by Doan, Racca and Magri (2022). In this paper, we theoretically interpret the transformations of an autoencoder. First, we remark that the latent space is a curved manifold with curvilinear coordinates, which can be analyzed with simple tools from Riemann geometry. Second, we characterize the geometrical properties of the latent space. We mathematically derive the metric tensor, which provides a mathematical description of the manifold. Third, we propose a method -- proper latent decomposition (PLD) -- that generalizes proper orthogonal decomposition of turbulent flows on the autoencoder latent space. This decomposition finds the dominant directions in the curved latent space. This theoretical work opens up computational opportunities for interpreting autoencoders and creating reduced-order models of turbulent flows.

Related papers

• Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness [32.929617755703696]
  Chain-of-thought (CoT) reasoning enables large language models to solve complex problems by generating intermediate reasoning steps.<n>We study the emphgeometry of CoT trajectories in the hidden state space of transformer models.<n>We show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions.<n> Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete.
  arXiv  Detail & Related papers  (2026-07-02T01:03:49Z)
• Random Dot Product Graphs as Dynamical Systems: Limitations and Opportunities [0.0]
  We investigate the differential equations governing the evolution of a temporal network within Random Dot Product Graphs.<n>We identify three fundamental obstructions: gauge freedom from rotational ambiguity in latent positions, realizability constraints from the manifold structure of the probability matrix, and trajectory recovery artifacts from spectral embedding.<n>We prove an identifiability principle: symmetric dynamics cannot absorb skew-symmetric gauge contamination, so dynamics structure can resolve obstructions.
  arXiv  Detail & Related papers  (2026-03-05T21:56:07Z)
• Riemannian Flow Matching for Disentangled Graph Domain Adaptation [51.98961391065951]
  Graph Domain Adaptation (GDA) typically uses adversarial learning to align graph embeddings in Euclidean space.<n>DisRFM is a geometry-aware GDA framework that unifies embedding and flow-based transport.
  arXiv  Detail & Related papers  (2026-01-31T11:05:35Z)
• Geodesic Calculus on Latent Spaces [4.023417156982924]
  We develop tools for a discrete Riemannian calculus approximating classical geometric operators.<n>We learn an approximate projection onto the latent manifold by minimizing a denoising objective.<n>We evaluate our approach on various autoencoders trained on synthetic and real data.
  arXiv  Detail & Related papers  (2025-10-10T15:25:03Z)
• Proper Latent Decomposition [4.266376725904727]
  We compute a reduced set of intrinsic coordinates (latent space) to accurately describe a flow with fewer degrees of freedom than the numerical discretization. With this proposed numerical framework, we propose an algorithm to perform PLD on the manifold. This work opens opportunities for analyzing autoencoders and latent spaces, nonlinear reduced-order modeling and scientific insights into the structure of high-dimensional data.
  arXiv  Detail & Related papers  (2024-12-01T12:19:08Z)
• Variational autoencoders with latent high-dimensional steady geometric flows for dynamics [0.0]
  We develop approaches to variational autoencoders (VAEs) for PDE-type ambient data with regularizing geometric latent dynamics. We redevelop the VAE framework such that manifold geometries, subject to our geometric flow, are learned in the intermediary latent space developed by encoders and decoders. We demonstrate, on our datasets of interest, our methods perform at least as well as the traditional VAE, and oftentimes better.
  arXiv  Detail & Related papers  (2024-10-14T04:07:45Z)
• Decoder Decomposition for the Analysis of the Latent Space of Nonlinear Autoencoders With Wind-Tunnel Experimental Data [3.7960472831772765]
  The goal of this paper is to propose a method to aid the interpretability of autoencoders. We propose the decoder decomposition, which is a post-processing method to connect the latent variables to the coherent structures of flows. The ability to rank and select latent variables will help users design and interpret nonlinear autoencoders.
  arXiv  Detail & Related papers  (2024-04-25T10:09:37Z)
• Topological Obstructions and How to Avoid Them [22.45861345237023]
  We show that local optima can arise due to singularities or an incorrect degree or winding number. We propose a new flow-based model that maps data points to multimodal distributions over geometric spaces.
  arXiv  Detail & Related papers  (2023-12-12T18:56:14Z)
• Alignment and Outer Shell Isotropy for Hyperbolic Graph Contrastive Learning [69.6810940330906]
  We propose a novel contrastive learning framework to learn high-quality graph embedding. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity which are related to properties of trees.
  arXiv  Detail & Related papers  (2023-10-27T15:31:42Z)
• Curve Your Attention: Mixed-Curvature Transformers for Graph Representation Learning [77.1421343649344]
  We propose a generalization of Transformers towards operating entirely on the product of constant curvature spaces. We also provide a kernelized approach to non-Euclidean attention, which enables our model to run in time and memory cost linear to the number of nodes and edges.
  arXiv  Detail & Related papers  (2023-09-08T02:44:37Z)
• Third quantization of open quantum systems: new dissipative symmetries and connections to phase-space and Keldysh field theory formulations [77.34726150561087]
  We reformulate the technique of third quantization in a way that explicitly connects all three methods. We first show that our formulation reveals a fundamental dissipative symmetry present in all quadratic bosonic or fermionic Lindbladians. For bosons, we then show that the Wigner function and the characteristic function can be thought of as ''wavefunctions'' of the density matrix.
  arXiv  Detail & Related papers  (2023-02-27T18:56:40Z)
• Convolutional autoencoder for the spatiotemporal latent representation of turbulence [5.8010446129208155]
  We employ a three-dimensional multiscale convolutional autoencoder (CAE) to obtain latent representation of a turbulent flow. We show that the Multiscale CAE is efficient, requiring less than 10% degrees of freedom than proper decomposition for compressing the data. The proposed deep learning architecture opens opportunities for nonlinear reduced-order modeling of turbulent flows from data.
  arXiv  Detail & Related papers  (2023-01-31T16:06:54Z)
• Modelling spatiotemporal turbulent dynamics with the convolutional autoencoder echo state network [5.8010446129208155]
  dynamics of turbulent flows is chaotic and difficult to predict. We propose a nonlinear decomposition of the turbulent state for a reduced-order representation of the dynamics.
  arXiv  Detail & Related papers  (2022-11-21T11:49:57Z)
• Shape And Structure Preserving Differential Privacy [70.08490462870144]
  We show how the gradient of the squared distance function offers better control over sensitivity than the Laplace mechanism. We also show how using the gradient of the squared distance function offers better control over sensitivity than the Laplace mechanism.
  arXiv  Detail & Related papers  (2022-09-21T18:14:38Z)
• Estimating Koopman operators for nonlinear dynamical systems: a nonparametric approach [77.77696851397539]
  The Koopman operator is a mathematical tool that allows for a linear description of non-linear systems. In this paper we capture their core essence as a dual version of the same framework, incorporating them into the Kernel framework. We establish a strong link between kernel methods and Koopman operators, leading to the estimation of the latter through Kernel functions.
  arXiv  Detail & Related papers  (2021-03-25T11:08:26Z)
• Manifold Learning via Manifold Deflation [105.7418091051558]
  dimensionality reduction methods provide a valuable means to visualize and interpret high-dimensional data. Many popular methods can fail dramatically, even on simple two-dimensional Manifolds. This paper presents an embedding method for a novel, incremental tangent space estimator that incorporates global structure as coordinates. Empirically, we show our algorithm recovers novel and interesting embeddings on real-world and synthetic datasets.
  arXiv  Detail & Related papers  (2020-07-07T10:04:28Z)
• Disentangling by Subspace Diffusion [72.1895236605335]
  We show that fully unsupervised factorization of a data manifold is possible if the true metric of the manifold is known. Our work reduces the question of whether unsupervised metric learning is possible, providing a unifying insight into the geometric nature of representation learning.
  arXiv  Detail & Related papers  (2020-06-23T13:33:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.