Invariant Representations via Wasserstein Correlation Maximization

• URL: http://arxiv.org/abs/2505.11702v1
• Date: Fri, 16 May 2025 21:11:51 GMT
• Title: Invariant Representations via Wasserstein Correlation Maximization
• Authors: Keenan Eikenberry, Lizuo Liu, Yoonsang Lee,
• Abstract summary: This work investigates the use of Wasserstein correlation for unsupervised representation learning.<n>We find that an (auto)encoder trained to maximize correlation between the input and distributions instead acts as a compressor.<n>We also establish various theoretical results for optimal transport-based dependence measures.
• Score: 0.6554326244334866
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work investigates the use of Wasserstein correlation -- a normalized measure of statistical dependence based on the Wasserstein distance between a joint distribution and the product of its marginals -- for unsupervised representation learning. Unlike, for example, contrastive methods, which naturally cluster classes in the latent space, we find that an (auto)encoder trained to maximize Wasserstein correlation between the input and encoded distributions instead acts as a compressor, reducing dimensionality while approximately preserving the topological and geometric properties of the input distribution. More strikingly, we show that Wasserstein correlation maximization can be used to arrive at an (auto)encoder -- either trained from scratch, or else one that extends a frozen, pretrained model -- that is approximately invariant to a chosen augmentation, or collection of augmentations, and that still approximately preserves the structural properties of the non-augmented input distribution. To do this, we first define the notion of an augmented encoder using the machinery of Markov-Wasserstein kernels. When the maximization objective is then applied to the augmented encoder, as opposed to the underlying, deterministic encoder, the resulting model exhibits the desired invariance properties. Finally, besides our experimental results, which show that even simple feedforward networks can be imbued with invariants or can, alternatively, be used to impart invariants to pretrained models under this training process, we additionally establish various theoretical results for optimal transport-based dependence measures. Code is available at https://github.com/keenan-eikenberry/wasserstein_correlation_maximization .

Related papers

• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Variational autoencoder with weighted samples for high-dimensional non-parametric adaptive importance sampling [0.0]
  We extend the existing framework to the case of weighted samples by introducing a new objective function. In order to add flexibility to the model and to be able to learn multimodal distributions, we consider a learnable prior distribution. We exploit the proposed procedure in existing adaptive importance sampling algorithms to draw points from a target distribution and to estimate a rare event probability in high dimension.
  arXiv  Detail & Related papers  (2023-10-13T15:40:55Z)
• Differentiating Metropolis-Hastings to Optimize Intractable Densities [51.16801956665228]
  We develop an algorithm for automatic differentiation of Metropolis-Hastings samplers. We apply gradient-based optimization to objectives expressed as expectations over intractable target densities.
  arXiv  Detail & Related papers  (2023-06-13T17:56:02Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Low-rank Characteristic Tensor Density Estimation Part II: Compression and Latent Density Estimation [31.631861197477185]
  Learning generative probabilistic models is a core problem in machine learning. This paper proposes a joint dimensionality reduction and non-parametric density estimation framework. We demonstrate that the proposed model achieves very promising results on regression tasks, sampling, and anomaly detection.
  arXiv  Detail & Related papers  (2021-06-20T00:38:56Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)
• Unbiased Gradient Estimation for Variational Auto-Encoders using Coupled Markov Chains [34.77971292478243]
  The variational auto-encoder (VAE) is a deep latent variable model that has two neural networks in an autoencoder-like architecture. We develop a training scheme for VAEs by introducing unbiased estimators of the log-likelihood gradient. We show experimentally that VAEs fitted with unbiased estimators exhibit better predictive performance.
  arXiv  Detail & Related papers  (2020-10-05T08:11:55Z)
• Generative Model without Prior Distribution Matching [26.91643368299913]
  Variational Autoencoder (VAE) and its variations are classic generative models by learning a low-dimensional latent representation to satisfy some prior distribution. We propose to let the prior match the embedding distribution rather than imposing the latent variables to fit the prior.
  arXiv  Detail & Related papers  (2020-09-23T09:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.