Fast and Efficient MMD-based Fair PCA via Optimization over Stiefel Manifold

• URL: http://arxiv.org/abs/2109.11196v1
• Date: Thu, 23 Sep 2021 08:06:02 GMT
• Title: Fast and Efficient MMD-based Fair PCA via Optimization over Stiefel Manifold
• Authors: Junghyun Lee, Gwangsu Kim, Matt Olfat, Mark Hasegawa-Johnson, Chang D. Yoo
• Abstract summary: This paper defines fair principal component analysis (PCA) as minimizing the maximum discrepancy (MMD) between dimensionality-reduced conditional distributions. We provide optimality guarantees and explicitly show the theoretical effect in practical settings.
• Score: 41.58534159822546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper defines fair principal component analysis (PCA) as minimizing the maximum mean discrepancy (MMD) between dimensionality-reduced conditional distributions of different protected classes. The incorporation of MMD naturally leads to an exact and tractable mathematical formulation of fairness with good statistical properties. We formulate the problem of fair PCA subject to MMD constraints as a non-convex optimization over the Stiefel manifold and solve it using the Riemannian Exact Penalty Method with Smoothing (REPMS; Liu and Boumal, 2019). Importantly, we provide local optimality guarantees and explicitly show the theoretical effect of each hyperparameter in practical settings, extending previous results. Experimental comparisons based on synthetic and UCI datasets show that our approach outperforms prior work in explained variance, fairness, and runtime.

Related papers

• A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models [63.949883238901414]
  We present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.
  arXiv  Detail & Related papers  (2024-08-29T17:46:18Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model. Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Benchmarking Parameter Control Methods in Differential Evolution for Mixed-Integer Black-Box Optimization [0.0]
  Differential evolution (DE) generally requires parameter control methods (PCMs) for the scale factor and crossover rate. This paper benchmarks PCMs in DE on the mixed-integer black-box optimization benchmarking function (bbob-mixint) suite. Although the PCM of SHADE is state-of-the-art for numerical black-box optimization, our results show its poor performance for mixed-integer black-box optimization.
  arXiv  Detail & Related papers  (2024-04-04T08:54:15Z)
• On the Asymptotic Mean Square Error Optimality of Diffusion Models [10.72484143420088]
  Diffusion models (DMs) as generative priors have recently shown great potential for denoising tasks. This paper proposes a novel denoising strategy inspired by the structure of the MSE-optimal conditional mean (CME) The resulting DM-based denoiser can be conveniently employed using a pre-trained DM, being particularly fast by truncating reverse diffusion steps.
  arXiv  Detail & Related papers  (2024-03-05T13:25:44Z)
• CoVO-MPC: Theoretical Analysis of Sampling-based MPC and Optimal Covariance Design [8.943418808959494]
  We characterize the convergence property of a widely used sampling-based Model Predictive Path Integral Control (MPPI) method. We show that MPPI enjoys at least linear convergence rates when the optimization is quadratic, which covers time-varying LQR systems. Our theoretical analysis directly leads to a novel sampling-based MPC algorithm, CoVo-MPC. Empirically, CoVo-MPC significantly outperforms standard MPPI by 43-54% in both simulations and real-world quad agile control tasks.
  arXiv  Detail & Related papers  (2024-01-14T21:10:59Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Pseudo-Spherical Contrastive Divergence [119.28384561517292]
  We propose pseudo-spherical contrastive divergence (PS-CD) to generalize maximum learning likelihood of energy-based models. PS-CD avoids the intractable partition function and provides a generalized family of learning objectives.
  arXiv  Detail & Related papers  (2021-11-01T09:17:15Z)
• Exact and Approximation Algorithms for Sparse PCA [1.7640556247739623]
  This paper proposes two exact mixed-integer SDPs (MISDPs) We then analyze the theoretical optimality gaps of their continuous relaxation values and prove that they are stronger than that of the state-of-art one. Since off-the-shelf solvers, in general, have difficulty in solving MISDPs, we approximate SPCA with arbitrary accuracy by a mixed-integer linear program (MILP) of a similar size as MISDPs.
  arXiv  Detail & Related papers  (2020-08-28T02:07:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.