StablePCA: Learning Shared Representations across Multiple Sources via Minimax Optimization

• URL: http://arxiv.org/abs/2505.00940v1
• Date: Fri, 02 May 2025 00:53:39 GMT
• Title: StablePCA: Learning Shared Representations across Multiple Sources via Minimax Optimization
• Authors: Zhenyu Wang, Molei Liu, Jing Lei, Francis Bach, Zijian Guo,
• Abstract summary: When multisource high-dimensional data, a key objective is to extract low-dimensional feature representations that effectively approximate the original features across different sources.<n>We propose the Mirror Stable Principal ComponentPCA, a novel method for group distribution of multi-dimensional representations.<n>We demonstrate high-dimensional accuracy and efficiency in extracting robust low-dimensional representations across various finite-sample scenarios.
• Score: 8.525738573467732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When synthesizing multisource high-dimensional data, a key objective is to extract low-dimensional feature representations that effectively approximate the original features across different sources. Such general feature extraction facilitates the discovery of transferable knowledge, mitigates systematic biases such as batch effects, and promotes fairness. In this paper, we propose Stable Principal Component Analysis (StablePCA), a novel method for group distributionally robust learning of latent representations from high-dimensional multi-source data. A primary challenge in generalizing PCA to the multi-source regime lies in the nonconvexity of the fixed rank constraint, rendering the minimax optimization nonconvex. To address this challenge, we employ the Fantope relaxation, reformulating the problem as a convex minimax optimization, with the objective defined as the maximum loss across sources. To solve the relaxed formulation, we devise an optimistic-gradient Mirror Prox algorithm with explicit closed-form updates. Theoretically, we establish the global convergence of the Mirror Prox algorithm, with the convergence rate provided from the optimization perspective. Furthermore, we offer practical criteria to assess how closely the solution approximates the original nonconvex formulation. Through extensive numerical experiments, we demonstrate StablePCA's high accuracy and efficiency in extracting robust low-dimensional representations across various finite-sample scenarios.

Related papers

• Preference Optimization for Combinatorial Optimization Problems [54.87466279363487]
  Reinforcement Learning (RL) has emerged as a powerful tool for neural optimization, enabling models learns that solve complex problems without requiring expert knowledge.<n>Despite significant progress, existing RL approaches face challenges such as diminishing reward signals and inefficient exploration in vast action spaces.<n>We propose Preference Optimization, a novel method that transforms quantitative reward signals into qualitative preference signals via statistical comparison modeling.
  arXiv  Detail & Related papers  (2025-05-13T16:47:00Z)
• Direct Distributional Optimization for Provable Alignment of Diffusion Models [39.048284342436666]
  We introduce a novel alignment method for diffusion models from distribution optimization perspectives.<n>We first formulate the problem as a generic regularized loss minimization over probability distributions.<n>We enable sampling from the learned distribution by approximating its score function via Doob's $h$-transform technique.
  arXiv  Detail & Related papers  (2025-02-05T07:35:15Z)
• Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications [79.53938312089308]
  The MIDX-Sampler is a novel adaptive sampling strategy based on an inverted multi-index approach.<n>Our method is backed by rigorous theoretical analysis, addressing key concerns such as sampling bias, gradient bias, convergence rates, and generalization error bounds.
  arXiv  Detail & Related papers  (2025-01-15T04:09:21Z)
• Alternating minimization for square root principal component pursuit [2.449191760736501]
  We develop efficient algorithms for solving the square root principal component pursuit (SRPCP) problem.<n>Specifically, we propose a tuning-free alternating minimization (AltMin) algorithm, where each iteration involves subproblems enjoying closed-form optimal solutions.<n>We introduce techniques based on the variational formulation of the nuclear norm and Burer-Monteiro decomposition to further accelerate the AltMin method.
  arXiv  Detail & Related papers  (2024-12-31T14:43:50Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Low-Rank Extragradient Methods for Scalable Semidefinite Optimization [17.384717824118255]
  We focus on high-dimensional and plausible settings in which the problem admits a low-rank solution.<n>We provide several theoretical results proving that, under these circumstances, the well-known Extragradient method converges to a solution of the constrained optimization problem.
  arXiv  Detail & Related papers  (2024-02-14T10:48:00Z)
• Self-Supervised Dataset Distillation for Transfer Learning [77.4714995131992]
  We propose a novel problem of distilling an unlabeled dataset into a set of small synthetic samples for efficient self-supervised learning (SSL) We first prove that a gradient of synthetic samples with respect to a SSL objective in naive bilevel optimization is textitbiased due to randomness originating from data augmentations or masking. We empirically validate the effectiveness of our method on various applications involving transfer learning.
  arXiv  Detail & Related papers  (2023-10-10T10:48:52Z)
• Scalable Distributional Robustness in a Class of Non Convex Optimization with Guarantees [7.541571634887807]
  Distributionally robust optimization (DRO) has shown robustness in learning as well as sample based problems. We propose a mixed-integer clustering program (MISOCP) which does not scale enough to solve problems with real worldsets.
  arXiv  Detail & Related papers  (2022-05-31T09:07:01Z)
• Sinkhorn Distributionally Robust Optimization [18.46110328123008]
  Sinkhorn distance is a variant of Wasserstein distance based on entropic regularization.<n>We derive a convex programming dual reformulation for general nominal distributions, transport costs, and loss functions.
  arXiv  Detail & Related papers  (2021-09-24T12:40:48Z)
• Outlier-Robust Sparse Estimation via Non-Convex Optimization [73.18654719887205]
  We explore the connection between high-dimensional statistics and non-robust optimization in the presence of sparsity constraints. We develop novel and simple optimization formulations for these problems. As a corollary, we obtain that any first-order method that efficiently converges to station yields an efficient algorithm for these tasks.
  arXiv  Detail & Related papers  (2021-09-23T17:38:24Z)
• Complexity-Free Generalization via Distributionally Robust Optimization [4.313143197674466]
  We present an alternate route to obtain generalization bounds on the solution from distributionally robust optimization (DRO) Our DRO bounds depend on the ambiguity set geometry and its compatibility with the true loss function. Notably, when using maximum mean discrepancy as a DRO distance metric, our analysis implies, to the best of our knowledge, the first generalization bound in the literature that depends solely on the true loss function.
  arXiv  Detail & Related papers  (2021-06-21T15:19:52Z)
• Adaptive Sampling for Best Policy Identification in Markov Decision Processes [79.4957965474334]
  We investigate the problem of best-policy identification in discounted Markov Decision (MDPs) when the learner has access to a generative model. The advantages of state-of-the-art algorithms are discussed and illustrated.
  arXiv  Detail & Related papers  (2020-09-28T15:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.