Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior

• URL: http://arxiv.org/abs/2504.18455v1
• Date: Fri, 25 Apr 2025 16:07:39 GMT
• Title: Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior
• Authors: Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski,
• Abstract summary: We study the problem of distributed multi-view representation learning.<n>In this problem, $K$ agents observe each one distinct, possibly statistically correlated, view and independently extracts from it a suitable representation.
• Score: 14.453654853392619
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the problem of distributed multi-view representation learning. In this problem, $K$ agents observe each one distinct, possibly statistically correlated, view and independently extracts from it a suitable representation in a manner that a decoder that gets all $K$ representations estimates correctly the hidden label. In the absence of any explicit coordination between the agents, a central question is: what should each agent extract from its view that is necessary and sufficient for a correct estimation at the decoder? In this paper, we investigate this question from a generalization error perspective. First, we establish several generalization bounds in terms of the relative entropy between the distribution of the representations extracted from training and "test" datasets and a data-dependent symmetric prior, i.e., the Minimum Description Length (MDL) of the latent variables for all views and training and test datasets. Then, we use the obtained bounds to devise a regularizer; and investigate in depth the question of the selection of a suitable prior. In particular, we show and conduct experiments that illustrate that our data-dependent Gaussian mixture priors with judiciously chosen weights lead to good performance. For single-view settings (i.e., $K=1$), our experimental results are shown to outperform existing prior art Variational Information Bottleneck (VIB) and Category-Dependent VIB (CDVIB) approaches. Interestingly, we show that a weighted attention mechanism emerges naturally in this setting. Finally, for the multi-view setting, we show that the selection of the joint prior as a Gaussians product mixture induces a Gaussian mixture marginal prior for each marginal view and implicitly encourages the agents to extract and output redundant features, a finding which is somewhat counter-intuitive.

Related papers

• Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective [14.480713752871523]
  (K) clients have each (n) training samples generated independently according to a possibly different data distribution. We study the effect of discrepancy between the clients' data distributions on the generalization error of the aggregated model. It is shown that the bound gets smaller as the degree of data heterogeneity across clients gets higher.
  arXiv  Detail & Related papers  (2025-03-03T14:33:38Z)
• Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors [14.453654853392619]
  We establish in-expectation and tail bounds on the generalization error of representation learning type algorithms.<n>We propose a systematic approach to simultaneously learning a data-dependent Gaussian mixture prior and using it as a regularizer.
  arXiv  Detail & Related papers  (2025-02-21T15:43:31Z)
• Bayesian Joint Additive Factor Models for Multiview Learning [7.254731344123118]
  A motivating application arises in the context of precision medicine where multi-omics data are collected to correlate with clinical outcomes. We propose a joint additive factor regression model (JAFAR) with a structured additive design, accounting for shared and view-specific components. Prediction of time-to-labor onset from immunome, metabolome, and proteome data illustrates performance gains against state-of-the-art competitors.
  arXiv  Detail & Related papers  (2024-06-02T15:35:45Z)
• Graph Out-of-Distribution Generalization with Controllable Data Augmentation [51.17476258673232]
  Graph Neural Network (GNN) has demonstrated extraordinary performance in classifying graph properties. Due to the selection bias of training and testing data, distribution deviation is widespread. We propose OOD calibration to measure the distribution deviation of virtual samples.
  arXiv  Detail & Related papers  (2023-08-16T13:10:27Z)
• Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data [4.971690889257356]
  We introduce an adaptation of the alternating minimization-descent scheme proposed by Collins and Nayer and Vaswani. We show that vanilla alternating-minimization descent fails catastrophically even for iid, but mildly non-isotropic data. Our analysis unifies and generalizes prior work, and provides a flexible framework for a wider range of applications.
  arXiv  Detail & Related papers  (2023-08-08T17:56:20Z)
• Approximating Counterfactual Bounds while Fusing Observational, Biased and Randomised Data Sources [64.96984404868411]
  We address the problem of integrating data from multiple, possibly biased, observational and interventional studies. We show that the likelihood of the available data has no local maxima. We then show how the same approach can address the general case of multiple datasets.
  arXiv  Detail & Related papers  (2023-07-31T11:28:24Z)
• Data thinning for convolution-closed distributions [2.299914829977005]
  We propose data thinning, an approach for splitting an observation into two or more independent parts that sum to the original observation. We show that data thinning can be used to validate the results of unsupervised learning approaches.
  arXiv  Detail & Related papers  (2023-01-18T02:47:41Z)
• Optimal Regularization for a Data Source [8.38093977965175]
  It is common to augment criteria that enforce data fidelity with a regularizer that promotes quantity in the solution. In this paper we seek a systematic understanding of the power and the limitations of convex regularization.
  arXiv  Detail & Related papers  (2022-12-27T20:11:59Z)
• A Robust and Flexible EM Algorithm for Mixtures of Elliptical Distributions with Missing Data [71.9573352891936]
  This paper tackles the problem of missing data imputation for noisy and non-Gaussian data. A new EM algorithm is investigated for mixtures of elliptical distributions with the property of handling potential missing data. Experimental results on synthetic data demonstrate that the proposed algorithm is robust to outliers and can be used with non-Gaussian data.
  arXiv  Detail & Related papers  (2022-01-28T10:01:37Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)
• Semi-Supervised Learning with Normalizing Flows [54.376602201489995]
  FlowGMM is an end-to-end approach to generative semi supervised learning with normalizing flows. We show promising results on a wide range of applications, including AG-News and Yahoo Answers text data.
  arXiv  Detail & Related papers  (2019-12-30T17:36:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.