Unlocking TriLevel Learning with Level-Wise Zeroth Order Constraints: Distributed Algorithms and Provable Non-Asymptotic Convergence

• URL: http://arxiv.org/abs/2412.07138v1
• Date: Tue, 10 Dec 2024 02:57:35 GMT
• Title: Unlocking TriLevel Learning with Level-Wise Zeroth Order Constraints: Distributed Algorithms and Provable Non-Asymptotic Convergence
• Authors: Yang Jiao, Kai Yang, Chengtao Jian,
• Abstract summary: An effective distributed trilevel zeroth order learning framework DTZO is proposed in this work.<n> DTZO is versatile and can be adapted to a wide range of (grey-box) TLL problems with partial zeroth order constraints.<n>Extensive experiments have been conducted to demonstrate and validate the superior performance of the proposed DTZO.
• Score: 9.95894026392039
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Trilevel learning (TLL) found diverse applications in numerous machine learning applications, ranging from robust hyperparameter optimization to domain adaptation. However, existing researches primarily focus on scenarios where TLL can be addressed with first order information available at each level, which is inadequate in many situations involving zeroth order constraints, such as when black-box models are employed. Moreover, in trilevel learning, data may be distributed across various nodes, necessitating strategies to address TLL problems without centralizing data on servers to uphold data privacy. To this end, an effective distributed trilevel zeroth order learning framework DTZO is proposed in this work to address the TLL problems with level-wise zeroth order constraints in a distributed manner. The proposed DTZO is versatile and can be adapted to a wide range of (grey-box) TLL problems with partial zeroth order constraints. In DTZO, the cascaded polynomial approximation can be constructed without relying on gradients or sub-gradients, leveraging a novel cut, i.e., zeroth order cut. Furthermore, we theoretically carry out the non-asymptotic convergence rate analysis for the proposed DTZO in achieving the $\epsilon$-stationary point. Extensive experiments have been conducted to demonstrate and validate the superior performance of the proposed DTZO, e.g., it approximately achieves up to a 40$\%$ improvement in performance.

Related papers

• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture. Non-smooth regularization is often incorporated into machine learning tasks. We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Optimization Proxies using Limited Labeled Data and Training Time -- A Semi-Supervised Bayesian Neural Network Approach [2.943640991628177]
  Constrained optimization problems arise in various engineering system operations such as inventory management electric power grids. This work introduces a learning scheme using Bayesian Networks (BNNs) to solve constrained optimization problems under limited data and restricted model times. We show that the proposed learning method outperforms conventional BNN and deep neural network (DNN) architectures.
  arXiv  Detail & Related papers  (2024-10-04T02:10:20Z)
• Byzantine-resilient Federated Learning Employing Normalized Gradients on Non-IID Datasets [23.640506243685863]
  In practical federated learning (FLNGA) the presence of malicious attacks and data heterogeneity often introduces biases into the learning process. We propose a Normalized Gradient unit (Fed-M) model which normalizes uploaded local gradients to be before aggregation, achieving a time of $mathcalO(pM)$.
  arXiv  Detail & Related papers  (2024-08-18T16:50:39Z)
• Provably Convergent Federated Trilevel Learning [19.15328400013401]
  Trilevel learning, also called trilevel optimization (TLO), has been recognized as a powerful modelling tool for decision process. This paper proposes an asynchronous federated trilevel optimization method to solve TLO problems.
  arXiv  Detail & Related papers  (2023-12-19T04:03:47Z)
• Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants [166.916517335816]
  In this paper, we offer a unified solution to the misalignment dilemma in the three tasks. We propose neural collapse terminus that is a fixed structure with the maximal equiangular inter-class separation for the whole label space. Our method holds the neural collapse optimality in an incremental fashion regardless of data imbalance or data scarcity.
  arXiv  Detail & Related papers  (2023-08-03T13:09:59Z)
• Modeling Inter-Class and Intra-Class Constraints in Novel Class Discovery [20.67503042774617]
  Novel class discovery (NCD) aims at learning a model that transfers the common knowledge from a class-disjoint labelled dataset to another unlabelled dataset. We propose to model both inter-class and intra-class constraints in NCD based on the symmetric Kullback-Leibler divergence (sKLD)
  arXiv  Detail & Related papers  (2022-10-07T14:46:32Z)
• Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods [19.587273175563745]
  Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. This paper proposes a unifying framework under the helm of spectral manifold learning to address those limitations.
  arXiv  Detail & Related papers  (2022-05-23T17:59:32Z)
• Truncated tensor Schatten p-norm based approach for spatiotemporal traffic data imputation with complicated missing patterns [77.34726150561087]
  We introduce four complicated missing patterns, including missing and three fiber-like missing cases according to the mode-drivenn fibers. Despite nonity of the objective function in our model, we derive the optimal solutions by integrating alternating data-mputation method of multipliers.
  arXiv  Detail & Related papers  (2022-05-19T08:37:56Z)
• Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Minimax Problems [80.46370778277186]
  Large scale convex-concave minimax problems arise in numerous applications, including game theory, robust training, and training of generative adversarial networks. We develop a communication-efficient distributed extragrad algorithm, LocalAdaSient, with an adaptive learning rate suitable for solving convex-concave minimax problem in the. Server model. We demonstrate its efficacy through several experiments in both the homogeneous and heterogeneous settings.
  arXiv  Detail & Related papers  (2021-06-18T09:42:05Z)
• Chance-Constrained Control with Lexicographic Deep Reinforcement Learning [77.34726150561087]
  This paper proposes a lexicographic Deep Reinforcement Learning (DeepRL)-based approach to chance-constrained Markov Decision Processes. A lexicographic version of the well-known DeepRL algorithm DQN is also proposed and validated via simulations.
  arXiv  Detail & Related papers  (2020-10-19T13:09:14Z)
• A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singleton [49.23948907229656]
  Bi-level Descent Aggregation is a flexible and modularized algorithmic framework for generic bi-level optimization. We derive a new methodology to prove the convergence of BDA without the LLS condition. Our investigations also demonstrate that BDA is indeed compatible to a verify of particular first-order computation modules.
  arXiv  Detail & Related papers  (2020-06-07T05:18:50Z)
• Generalized Zero-Shot Learning Via Over-Complete Distribution [79.5140590952889]
  We propose to generate an Over-Complete Distribution (OCD) using Conditional Variational Autoencoder (CVAE) of both seen and unseen classes. The effectiveness of the framework is evaluated using both Zero-Shot Learning and Generalized Zero-Shot Learning protocols.
  arXiv  Detail & Related papers  (2020-04-01T19:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.