Robustness of Nonlinear Representation Learning

• URL: http://arxiv.org/abs/2503.15355v1
• Date: Wed, 19 Mar 2025 15:57:03 GMT
• Title: Robustness of Nonlinear Representation Learning
• Authors: Simon Buchholz, Bernhard Schölkopf,
• Abstract summary: We study the problem of unsupervised representation learning in slightly misspecified settings.<n>We show that the mixing can be identified up to linear transformations and small errors.<n>Those results are a step towards identifiability results for unsupervised representation learning for real-world data.
• Score: 60.15898117103069
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the problem of unsupervised representation learning in slightly misspecified settings, and thus formalize the study of robustness of nonlinear representation learning. We focus on the case where the mixing is close to a local isometry in a suitable distance and show based on existing rigidity results that the mixing can be identified up to linear transformations and small errors. In a second step, we investigate Independent Component Analysis (ICA) with observations generated according to $x=f(s)=As+h(s)$ where $A$ is an invertible mixing matrix and $h$ a small perturbation. We show that we can approximately recover the matrix $A$ and the independent components. Together, these two results show approximate identifiability of nonlinear ICA with almost isometric mixing functions. Those results are a step towards identifiability results for unsupervised representation learning for real-world data that do not follow restrictive model classes.

Related papers

• Adversarial Dependence Minimization [78.36795688238155]
  This work provides a differentiable and scalable algorithm for dependence minimization that goes beyond linear pairwise decorrelation.<n>We demonstrate its utility in three applications: extending PCA to nonlinear decorrelation, improving the generalization of image classification methods, and preventing dimensional collapse in self-supervised representation learning.
  arXiv  Detail & Related papers  (2025-02-05T14:43:40Z)
• Understanding Self-Supervised Learning via Gaussian Mixture Models [19.51336063093898]
  We analyze self-supervised learning in a natural context: dimensionality reduction in Gaussian Mixture Models.<n>We show that vanilla contrastive learning is able to find the optimal lower-dimensional subspace even when the Gaussians are not isotropic.<n>In this setting we show that contrastive learning learns the subset of fisher-optimal subspace, effectively filtering out all the noise from the learnt representations.
  arXiv  Detail & Related papers  (2024-11-05T21:43:05Z)
• On Characterizing and Mitigating Imbalances in Multi-Instance Partial Label Learning [57.18649648182171]
  We make contributions towards addressing a problem that hasn't been studied so far in the context of MI-PLL.<n>We derive class-specific risk bounds for MI-PLL, while making minimal assumptions.<n>Our theory reveals a unique phenomenon: that $sigma$ can greatly impact learning imbalances.
  arXiv  Detail & Related papers  (2024-07-13T20:56:34Z)
• Reinforcement Learning with Function Approximation: From Linear to Nonlinear [4.314956204483073]
  This paper reviews recent results on error analysis for reinforcement learning algorithms in linear or nonlinear approximation settings. We discuss various properties related to approximation error and present concrete conditions on transition probability and reward function.
  arXiv  Detail & Related papers  (2023-02-20T00:31:18Z)
• Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data [19.72282903349282]
  We show a general class of nonlinear loss functions for multimodal contrastive learning (MMCL) We quantitatively show that the feature learning ability of MMCL can be better than that of unimodal contrastive learning applied to each modality. When we have access to additional unpaired data, we propose a new MMCL loss that incorporates additional unpaired datasets.
  arXiv  Detail & Related papers  (2023-02-13T10:11:05Z)
• Provable Subspace Identification Under Post-Nonlinear Mixtures [11.012445089716016]
  Untrivial mixture learning aims at identifying linearly or nonlinearly mixed latent components in a blind manner. This work shows that under a carefully designed criterion, a null space associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity.
  arXiv  Detail & Related papers  (2022-10-14T05:26:40Z)
• Functional Nonlinear Learning [0.0]
  We propose a functional nonlinear learning (FunNoL) method to represent multivariate functional data in a lower-dimensional feature space. We show that FunNoL provides satisfactory curve classification and reconstruction regardless of data sparsity.
  arXiv  Detail & Related papers  (2022-06-22T23:47:45Z)
• Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systems [45.17023170054112]
  We consider the setting of vector valued non-linear dynamical systems $X_t+1 = phi(A* X_t) + eta_t$, where $eta_t$ is unbiased noise and $phi : mathbbR to mathbbR$ is a known link function that satisfies certain em expansivity property.
  arXiv  Detail & Related papers  (2021-05-24T22:14:26Z)
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation [63.042651834453544]
  We show that the unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data. We observe that while the different methods successfully enforce properties "encouraged" by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision.
  arXiv  Detail & Related papers  (2020-10-27T10:17:15Z)
• Learning Mixtures of Low-Rank Models [89.39877968115833]
  We study the problem of learning computational mixtures of low-rank models. We develop an algorithm that is guaranteed to recover the unknown matrices with near-optimal sample. In addition, the proposed algorithm is provably stable against random noise.
  arXiv  Detail & Related papers  (2020-09-23T17:53:48Z)
• Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees [106.91654068632882]
  We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution. We show that the proposed objective is strongly convex in a neighborhood around the ground truth, so that a gradient descent-based method achieves linear convergence rate. Our estimator is robust to any model misspecification within the exponential family, which is validated in extensive experiments.
  arXiv  Detail & Related papers  (2020-03-02T16:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.