Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization

• URL: http://arxiv.org/abs/2509.18997v2
• Date: Thu, 02 Oct 2025 13:30:05 GMT
• Title: Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization
• Authors: Pascal Esser, Maximilian Fleissner, Debarghya Ghoshdastidar,
• Abstract summary: Representation learning from unlabeled data has been extensively studied in statistics, data science and signal processing.<n>Current deep learning models use new principles for unsupervised representation learning that cannot be easily analyzed using classical theories.<n>Visual foundation models have found tremendous success using self-supervision or denoising/masked autoencoders.
• Score: 9.69858219543662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representation learning from unlabeled data has been extensively studied in statistics, data science and signal processing with a rich literature on techniques for dimension reduction, compression, multi-dimensional scaling among others. However, current deep learning models use new principles for unsupervised representation learning that cannot be easily analyzed using classical theories. For example, visual foundation models have found tremendous success using self-supervision or denoising/masked autoencoders, which effectively learn representations from massive amounts of unlabeled data. However, it remains difficult to characterize the representations learned by these models and to explain why they perform well for diverse prediction tasks or show emergent behavior. To answer these questions, one needs to combine mathematical tools from statistics and optimization. This paper provides an overview of recent theoretical advances in representation learning from unlabeled data and mentions our contributions in this direction.

Related papers

• When and How Unlabeled Data Provably Improve In-Context Learning [31.201385551730926]
  In-supervised learning can be effective even when demonstrations have missing or incorrect labels.<n>We show that multilayer or looped transformers can effectively leverage unlabeled data by implicitly constructing estimators of the form $sum_ige 0 a_i (Xtop X)iXtop y$ with $X$ and $y$ denoting features and partially-observed labels.
  arXiv  Detail & Related papers  (2025-06-18T10:01:17Z)
• Learning Latent Graph Structures and their Uncertainty [63.95971478893842]
  We show that minimizing point-prediction losses does not guarantee proper learning of latent relational information.<n>We propose a sampling-based method that solves this joint learning task.
  arXiv  Detail & Related papers  (2024-05-30T10:49:22Z)
• Learning Defect Prediction from Unrealistic Data [57.53586547895278]
  Pretrained models of code have become popular choices for code understanding and generation tasks. Such models tend to be large and require commensurate volumes of training data. It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs. Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
  arXiv  Detail & Related papers  (2023-11-02T01:51:43Z)
• The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning [32.15608637930748]
  We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously. We provide analysis using a theoretical data model and show that, while more diverse pre-training data result in more diverse features for different tasks, it puts less emphasis on task-specific features.
  arXiv  Detail & Related papers  (2023-02-28T22:14:33Z)
• A Survey of Learning on Small Data: Generalization, Optimization, and Challenge [101.27154181792567]
  Learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI. This survey follows the active sampling theory under a PAC framework to analyze the generalization error and label complexity of learning on small data. Multiple data applications that may benefit from efficient small data representation are surveyed.
  arXiv  Detail & Related papers  (2022-07-29T02:34:19Z)
• Relation-Guided Representation Learning [53.60351496449232]
  We propose a new representation learning method that explicitly models and leverages sample relations. Our framework well preserves the relations between samples. By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
  arXiv  Detail & Related papers  (2020-07-11T10:57:45Z)
• Pseudo-Representation Labeling Semi-Supervised Learning [0.0]
  In recent years, semi-supervised learning has shown tremendous success in leveraging unlabeled data to improve the performance of deep learning models. This work proposes the pseudo-representation labeling, a simple and flexible framework that utilizes pseudo-labeling techniques to iteratively label a small amount of unlabeled data and use them as training data. Compared with the existing approaches, the pseudo-representation labeling is more intuitive and can effectively solve practical problems in the real world.
  arXiv  Detail & Related papers  (2020-05-31T03:55:41Z)
• Out-of-Sample Representation Learning for Multi-Relational Graphs [8.956321788625894]
  We study the out-of-sample representation learning problem for non-attributed knowledge graphs. We create benchmark datasets for this task, develop several models and baselines, and provide empirical analyses and comparisons of the proposed models and baselines.
  arXiv  Detail & Related papers  (2020-04-28T00:53:01Z)
• Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision [57.14468881854616]
  We propose an auxiliary training objective that improves the generalization capabilities of neural networks. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
  arXiv  Detail & Related papers  (2020-04-20T02:47:49Z)
• Laplacian Denoising Autoencoder [114.21219514831343]
  We propose to learn data representations with a novel type of denoising autoencoder. The noisy input data is generated by corrupting latent clean data in the gradient domain. Experiments on several visual benchmarks demonstrate that better representations can be learned with the proposed approach.
  arXiv  Detail & Related papers  (2020-03-30T16:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.