Fair Principal Component Analysis and Filter Design

• URL: http://arxiv.org/abs/2002.06557v2
• Date: Tue, 1 Jun 2021 19:10:20 GMT
• Title: Fair Principal Component Analysis and Filter Design
• Authors: Gad Zalcberg and Ami Wiesel
• Abstract summary: We search for a low dimensional subspace that spans multiple target vectors in a fair manner. We analyze the landscape of the underlying optimization in the case of targets. We prove that the landscape is benign and that all local minima are globally optimal.
• Score: 2.66512000865131
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider Fair Principal Component Analysis (FPCA) and search for a low dimensional subspace that spans multiple target vectors in a fair manner. FPCA is defined as a non-concave maximization of the worst projected target norm within a given set. The problem arises in filter design in signal processing, and when incorporating fairness into dimensionality reduction schemes. The state of the art approach to FPCA is via semidefinite relaxation and involves a polynomial yet computationally expensive optimization. To allow scalability, we propose to address FPCA using naive sub-gradient descent. We analyze the landscape of the underlying optimization in the case of orthogonal targets. We prove that the landscape is benign and that all local minima are globally optimal. Interestingly, the SDR approach leads to sub-optimal solutions in this simple case. Finally, we discuss the equivalence between orthogonal FPCA and the design of normalized tight frames.

Related papers

• LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models [48.68246945083386]
  Likelihood-Free Policy Optimization (LFPO) is a native framework that maps the concept of vector field flow matching to the discrete token space.<n>LFPO formulates alignment as geometric velocity rectification, which directly optimize denoising logits via contrastive updates.<n>Experiments demonstrate that LFPO not only outperforms state-of-the-art baselines on code and reasoning benchmarks but also accelerates inference by approximately 20% through reduced diffusion steps.
  arXiv  Detail & Related papers  (2026-03-02T07:42:55Z)
• Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control [55.366871033602145]
  We argue that robustness failures cannot be addressed by data-centric methods alone.<n>We propose ShaPO, a geometry-aware preference optimization framework.<n>ShaPO enforces worst-case alignment objectives via selective geometry control over alignment-critical parameter subspace.
  arXiv  Detail & Related papers  (2026-02-07T03:46:33Z)
• Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulations [121.39938773554523]
  The Area Under the ROC Curve (AUC) is a pivotal evaluation metric in real-world scenarios with both class imbalance and decision constraints.<n>We present two simple instance-wise minimax reformulations to close the approximation gap of PAUC optimization.<n>The resulting algorithms enjoy a linear per-iteration computational complexity w.r.t. the sample size and a convergence rate of $O(-2/3)$ for typical one-way and two-way PAUCs.
  arXiv  Detail & Related papers  (2025-12-01T02:52:33Z)
• From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models [90.45197506653341]
  Large reasoning models generate intermediate reasoning traces before producing final answers.<n> aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored.<n>A common workaround optimized a single sampled trajectory, which introduces substantial gradient variance from trace sampling.
  arXiv  Detail & Related papers  (2025-10-06T17:58:01Z)
• Hidden Convexity of Fair PCA and Fast Solver via Eigenvalue Optimization [1.4999444543328293]
  Principal Component Analysis (PCA) is a technique in machine learning for dimensionality reduction of high-dimensional datasets. The Fair (FPCA) model was introduced by Samadi et al. to equalize the reconstruction loss between subgroups. The semidefinite relaxation (SDR) based approach proposed by Samadi et al. is computationally expensive even for suboptimal solutions. In this paper, we identify a hidden convexity in the FPCA model and introduce an algorithm for convex optimization via eigenvalue optimization.
  arXiv  Detail & Related papers  (2025-03-01T02:13:20Z)
• Tuning-Free Structured Sparse PCA via Deep Unfolding Networks [5.931547772157972]
  We propose a new type of sparse principal component analysis (PCA) for unsupervised feature selection (UFS) We use an interpretable deep unfolding network that translates iterative optimization steps into trainable neural architectures. This innovation enables automatic learning of the regularization parameters, effectively bypassing the empirical tuning requirements of conventional methods.
  arXiv  Detail & Related papers  (2025-02-28T08:32:51Z)
• Preference-Optimized Pareto Set Learning for Blackbox Optimization [1.9628841617148691]
  No single solution exists that can optimize all the objectives simultaneously. In a typical MOO problem, the goal is to find a set of optimum solutions (Pareto set) that trades off the preferences among objectives. Our formulation leads to a bilevel optimization problem that can be solved by e.g. differentiable cross-entropy methods.
  arXiv  Detail & Related papers  (2024-08-19T13:23:07Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model. Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback [106.63518036538163]
  We present a novel unified bilevel optimization-based framework, textsfPARL, formulated to address the recently highlighted critical issue of policy alignment in reinforcement learning. Our framework addressed these concerns by explicitly parameterizing the distribution of the upper alignment objective (reward design) by the lower optimal variable. Our empirical results substantiate that the proposed textsfPARL can address the alignment concerns in RL by showing significant improvements.
  arXiv  Detail & Related papers  (2023-08-03T18:03:44Z)
• Prescriptive PCA: Dimensionality Reduction for Two-stage Stochastic Optimization [1.1612308609123565]
  We develop a prescriptive dimensionality reduction framework that aims to minimize the degree of suboptimality in the optimization phase. For the case where the downstream optimization problem has an expected value objective, we show that prescriptive dimensionality reduction can be performed via solving a distributionally-robust optimization problem. Our approach significantly outperforms principal component analysis with real and synthetic data sets.
  arXiv  Detail & Related papers  (2023-06-04T00:50:35Z)
• SING: A Plug-and-Play DNN Learning Technique [25.563053353709627]
  We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and robustness of Adam(W) SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of gradients fed to Adam(W)
  arXiv  Detail & Related papers  (2023-05-25T12:39:45Z)
• Nesterov Meets Optimism: Rate-Optimal Separable Minimax Optimization [108.35402316802765]
  We propose a new first-order optimization algorithm -- AcceleratedGradient-OptimisticGradient (AG-OG) Ascent. We show that AG-OG achieves the optimal convergence rate (up to a constant) for a variety of settings. We extend our algorithm to extend the setting and achieve the optimal convergence rate in both bi-SC-SC and bi-C-SC settings.
  arXiv  Detail & Related papers  (2022-10-31T17:59:29Z)
• Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks [12.90962626557934]
  Conal neural networks (CNNs) have shown very appealing performance for many computer vision applications. In this paper, we propose a moment centralization-based SGD datasets for CNNs. The proposed moment centralization is generic in nature and can be integrated with any of the existing adaptive momentum-baseds.
  arXiv  Detail & Related papers  (2022-07-19T04:38:01Z)
• Large-scale Optimization of Partial AUC in a Range of False Positive Rates [51.12047280149546]
  The area under the ROC curve (AUC) is one of the most widely used performance measures for classification models in machine learning. We develop an efficient approximated gradient descent method based on recent practical envelope smoothing technique. Our proposed algorithm can also be used to minimize the sum of some ranked range loss, which also lacks efficient solvers.
  arXiv  Detail & Related papers  (2022-03-03T03:46:18Z)
• Fast Batch Nuclear-norm Maximization and Minimization for Robust Domain Adaptation [154.2195491708548]
  We study the prediction discriminability and diversity by studying the structure of the classification output matrix of a randomly selected data batch. We propose Batch Nuclear-norm Maximization and Minimization, which performs nuclear-norm on the target output matrix to enhance the target prediction ability. Experiments show that our method could boost the adaptation accuracy and robustness under three typical domain adaptation scenarios.
  arXiv  Detail & Related papers  (2021-07-13T15:08:32Z)
• Discretization-Aware Architecture Search [81.35557425784026]
  This paper presents discretization-aware architecture search (DAtextsuperscript2S) The core idea is to push the super-network towards the configuration of desired topology, so that the accuracy loss brought by discretization is largely alleviated. Experiments on standard image classification benchmarks demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2020-07-07T01:18:58Z)
• Convergence of adaptive algorithms for weakly convex constrained optimization [59.36386973876765]
  We prove the $mathcaltilde O(t-1/4)$ rate of convergence for the norm of the gradient of Moreau envelope. Our analysis works with mini-batch size of $1$, constant first and second order moment parameters, and possibly smooth optimization domains.
  arXiv  Detail & Related papers  (2020-06-11T17:43:19Z)
• Objective-Sensitive Principal Component Analysis for High-Dimensional Inverse Problems [0.0]
  We present a novel approach for adaptive, differentiable parameterization of large-scale random fields. The developed technique is based on principal component analysis (PCA) but modifies a purely data-driven basis of principal components considering objective function behavior. Three algorithms for optimal parameter decomposition are presented and applied to an objective of 2D synthetic history matching.
  arXiv  Detail & Related papers  (2020-06-02T18:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.