D-SVM over Networked Systems with Non-Ideal Linking Conditions

• URL: http://arxiv.org/abs/2304.06667v1
• Date: Thu, 13 Apr 2023 16:56:57 GMT
• Title: D-SVM over Networked Systems with Non-Ideal Linking Conditions
• Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Houman Zarrabi
• Abstract summary: This paper considers distributed optimization algorithms, with application in binary classification via distributed support-vector-machines (D-SVM) The agents solve a consensus-constraint distributed optimization cooperatively via continuous-time dynamics, while the links are subject to strongly sign-preserving odd nonlinear conditions. Logarithmic quantization and clipping (saturation) are two examples of such nonlinearities.
• Score: 5.962184741057505
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper considers distributed optimization algorithms, with application in binary classification via distributed support-vector-machines (D-SVM) over multi-agent networks subject to some link nonlinearities. The agents solve a consensus-constraint distributed optimization cooperatively via continuous-time dynamics, while the links are subject to strongly sign-preserving odd nonlinear conditions. Logarithmic quantization and clipping (saturation) are two examples of such nonlinearities. In contrast to existing literature that mostly considers ideal links and perfect information exchange over linear channels, we show how general sector-bounded models affect the convergence to the optimizer (i.e., the SVM classifier) over dynamic balanced directed networks. In general, any odd sector-bounded nonlinear mapping can be applied to our dynamics. The main challenge is to show that the proposed system dynamics always have one zero eigenvalue (associated with the consensus) and the other eigenvalues all have negative real parts. This is done by recalling arguments from matrix perturbation theory. Then, the solution is shown to converge to the agreement state under certain conditions. For example, the gradient tracking (GT) step size is tighter than the linear case by factors related to the upper/lower sector bounds. To the best of our knowledge, no existing work in distributed optimization and learning literature considers non-ideal link conditions.

Related papers

• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures [14.93489065234423]
  We present a general variational framework for the training of freeform nonlinearities in layered computational architectures. The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. We show how to solve the numerically function-optimization problem by representing the nonlinearities in a suitable (nonuniform) B-spline basis.
  arXiv  Detail & Related papers  (2024-08-23T14:39:27Z)
• Cons-training tensor networks [2.8834278113855896]
  We introduce a novel family of tensor networks, termed. textitconstrained matrix product states (MPS) These networks incorporate exactly arbitrary discrete linear constraints, including inequalities, into sparse block structures. These networks are particularly tailored for modeling distributions with support strictly over the feasible space.
  arXiv  Detail & Related papers  (2024-05-15T00:13:18Z)
• Discretized Distributed Optimization over Dynamic Digraphs [6.239775988430136]
  We consider a discrete-time model of continuous-time distributed optimization over dynamic directed-graphs (digraphs) Our algorithm works over general strongly connected dynamic networks under switching topologies. The proposed framework finds applications to distributed classification and learning.
  arXiv  Detail & Related papers  (2023-11-14T06:33:41Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Optimization-Induced Graph Implicit Nonlinear Diffusion [64.39772634635273]
  We propose a new kind of graph convolution variants, called Graph Implicit Diffusion (GIND) GIND implicitly has access to infinite hops of neighbors while adaptively aggregating features with nonlinear diffusion to prevent over-smoothing. We show that the learned representation can be formalized as the minimizer of an explicit convex optimization objective.
  arXiv  Detail & Related papers  (2022-06-29T06:26:42Z)
• Exact solutions of interacting dissipative systems via weak symmetries [77.34726150561087]
  We analytically diagonalize the Liouvillian of a class Markovian dissipative systems with arbitrary strong interactions or nonlinearity. This enables an exact description of the full dynamics and dissipative spectrum. Our method is applicable to a variety of other systems, and could provide a powerful new tool for the study of complex driven-dissipative quantum systems.
  arXiv  Detail & Related papers  (2021-09-27T17:45:42Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Optimization on manifolds: A symplectic approach [127.54402681305629]
  We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
  arXiv  Detail & Related papers  (2021-07-23T13:43:34Z)
• Towards Understanding Generalization via Decomposing Excess Risk Dynamics [13.4379473119565]
  We analyze the generalization dynamics to derive algorithm-dependent bounds, e.g., stability. Inspired by the observation that neural networks show a slow convergence rate when fitting noise, we propose decomposing the excess risk dynamics. Under the decomposition framework, the new bound accords better with the theoretical and empirical evidence compared to the stability-based bound and uniform convergence bound.
  arXiv  Detail & Related papers  (2021-06-11T03:42:45Z)
• Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems [107.3868459697569]
  We introduce an eigendecomposition-free approach to training a deep network. We show that our approach is much more robust than explicit differentiation of the eigendecomposition. Our method has better convergence properties and yields state-of-the-art results.
  arXiv  Detail & Related papers  (2020-04-15T04:29:34Z)
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks [20.44438519046223]
  We show that wide neural networks satisfy the PL$*$ condition, which explains the (S)GD convergence to a global minimum. We show that wide neural networks satisfy the PL$*$ condition, which explains the (S)GD convergence to a global minimum.
  arXiv  Detail & Related papers  (2020-02-29T17:18:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.