Understanding Self-Supervised Learning via Latent Distribution Matching

• URL: http://arxiv.org/abs/2605.03517v1
• Date: Tue, 05 May 2026 08:53:00 GMT
• Title: Understanding Self-Supervised Learning via Latent Distribution Matching
• Abstract summary: Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones.<n>We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment)<n>This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches.
• Score: 8.228889210180268
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.

Related papers

• Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine [60.669081685261965]
  Diffusion models generate high-dimensional data with remarkable quality.<n>How their training efficiently learns the score function remains theoretically unexplained.<n>We instantiate this principle as Score-induced Latent Diffusion (SiLD)
  arXiv  Detail & Related papers  (2026-05-16T16:51:10Z)
• LEPO: Latent Reasoning Policy Optimization for Large Language Models [11.032175358561162]
  We introduce controllableity into latent reasoning via Gumbel-Softmax.<n>We propose textbfunderlineLatent Rtextbfunderlineeasoning textbfunderlinePolicy textbfunderlineOptimization(textbfLEPO)<n>In tests, LEPO significantly outperforms existing RL methods for discrete and latent reasoning.
  arXiv  Detail & Related papers  (2026-04-20T07:05:12Z)
• Latent Nonlinear Denoising Score Matching for Enhanced Learning of Structured Distributions [3.060720241524644]
  We present latent nonlinear denoising score matching (LNDSM)<n>LNDSM is a novel training objective for score-based generative models that integrates nonlinear forward dynamics with the VAE-based latent SGM framework.<n> Experiments on variants of the MNIST dataset demonstrate that the proposed method achieves faster synthesis and enhanced learning of inherently structured distributions.
  arXiv  Detail & Related papers  (2025-12-07T01:17:14Z)
• Maximum Likelihood Learning of Latent Dynamics Without Reconstruction [9.186673054867864]
  We introduce a novel unsupervised learning method for time series data with latent dynamical structure: the recognition-parametrized Gaussian state space model (RP-GSSM)<n>The RP-GSSM is a probabilistic model that learns Markovian Gaussian latents explaining statistical dependence between observations at different time steps.<n>We show how this approach outperforms alternatives on problems that include learning nonlinear dynamics from video, with or without background distractors.
  arXiv  Detail & Related papers  (2025-05-29T15:44:20Z)
• Hybrid Latent Reasoning via Reinforcement Learning [50.6763762323985]
  We explore latent reasoning by leveraging the capabilities of large language models (LLMs) via reinforcement learning (RL)<n>We introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid latent reasoning approach that integrates prior hidden states into sampled tokens with a learnable gating mechanism.<n>HRPO-trained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths.
  arXiv  Detail & Related papers  (2025-05-24T01:26:16Z)
• Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models [104.55763564037831]
  We train a regression model that leverages attention maps, probabilities on the current generation step, and recurrently computed uncertainty scores from previously generated tokens.<n>Our evaluation shows that the proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.
  arXiv  Detail & Related papers  (2024-08-20T09:42:26Z)
• The Common Stability Mechanism behind most Self-Supervised Learning Approaches [64.40701218561921]
  We provide a framework to explain the stability mechanism of different self-supervised learning techniques. We discuss the working mechanism of contrastive techniques like SimCLR, non-contrastive techniques like BYOL, SWAV, SimSiam, Barlow Twins, and DINO. We formulate different hypotheses and test them using the Imagenet100 dataset.
  arXiv  Detail & Related papers  (2024-02-22T20:36:24Z)
• Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? [35.15844215216846]
  EDL methods are trained to learn a meta distribution over the predictive distribution by minimizing a specific objective function. Recent studies identify limitations of the existing methods to conclude their learned uncertainties are unreliable. We provide a sharper understanding of the behavior of a wide class of EDL methods by unifying various objective functions. We conclude that even when EDL methods are empirically effective on downstream tasks, this occurs despite their poor uncertainty quantification capabilities.
  arXiv  Detail & Related papers  (2024-02-09T03:23:39Z)
• Representation Learning Dynamics of Self-Supervised Models [7.289672463326423]
  Self-Supervised Learning (SSL) is an important paradigm for learning representations from unlabelled data. We study the learning dynamics of SSL models, specifically representations obtained by minimising contrastive and non-contrastive losses. We derive the exact learning dynamics of the SSL models trained using gradient descent on the Grassmannian manifold.
  arXiv  Detail & Related papers  (2023-09-05T07:48:45Z)
• Regularizing with Pseudo-Negatives for Continual Self-Supervised Learning [62.40718385934608]
  We introduce a novel Pseudo-Negative Regularization (PNR) framework for effective continual self-supervised learning (CSSL) Our PNR leverages pseudo-negatives obtained through model-based augmentation in a way that newly learned representations may not contradict what has been learned in the past.
  arXiv  Detail & Related papers  (2023-06-08T10:59:35Z)
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning [77.57285768500225]
  Recently proposed consistency-based Semi-Supervised Learning (SSL) methods have advanced the state of the art in several SSL tasks. Despite these advances, the understanding of these methods is still relatively limited.
  arXiv  Detail & Related papers  (2021-01-18T10:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.