Quantization for decentralized learning under subspace constraints
- URL: http://arxiv.org/abs/2209.07821v2
- Date: Fri, 28 Jul 2023 20:59:35 GMT
- Title: Quantization for decentralized learning under subspace constraints
- Authors: Roula Nassif, Stefan Vlaski, Marco Carpentiero, Vincenzo Matta, Marc
Antonini, Ali H. Sayed
- Abstract summary: We consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints.
We propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates.
The analysis shows that, under some general conditions on the quantization noise, the strategy is stable both in terms of mean-square error and average bit rate.
- Score: 61.59416703323886
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we consider decentralized optimization problems where agents
have individual cost functions to minimize subject to subspace constraints that
require the minimizers across the network to lie in low-dimensional subspaces.
This constrained formulation includes consensus or single-task optimization as
special cases, and allows for more general task relatedness models such as
multitask smoothness and coupled optimization. In order to cope with
communication constraints, we propose and study an adaptive decentralized
strategy where the agents employ differential randomized quantizers to compress
their estimates before communicating with their neighbors. The analysis shows
that, under some general conditions on the quantization noise, and for
sufficiently small step-sizes $\mu$, the strategy is stable both in terms of
mean-square error and average bit rate: by reducing $\mu$, it is possible to
keep the estimation errors small (on the order of $\mu$) without increasing
indefinitely the bit rate as $\mu\rightarrow 0$. Simulations illustrate the
theoretical findings and the effectiveness of the proposed approach, revealing
that decentralized learning is achievable at the expense of only a few bits.
Related papers
- Minimax and Communication-Efficient Distributed Best Subset Selection with Oracle Property [0.358439716487063]
The explosion of large-scale data has outstripped the processing capabilities of single-machine systems.
Traditional approaches to distributed inference often struggle with achieving true sparsity in high-dimensional datasets.
We propose a novel, two-stage, distributed best subset selection algorithm to address these issues.
arXiv Detail & Related papers (2024-08-30T13:22:08Z) - Differential error feedback for communication-efficient decentralized learning [48.924131251745266]
We propose a new decentralized communication-efficient learning approach that blends differential quantization with error feedback.
We show that the resulting communication-efficient strategy is stable both in terms of mean-square error and average bit rate.
The results establish that, in the small step-size regime and with a finite number of bits, it is possible to attain the performance achievable in the absence of compression.
arXiv Detail & Related papers (2024-06-26T15:11:26Z) - Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [94.2860766709971]
We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents.
Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
arXiv Detail & Related papers (2024-04-04T06:24:11Z) - Communication-Efficient Zeroth-Order Distributed Online Optimization:
Algorithm, Theory, and Applications [9.045332526072828]
This paper focuses on a multi-agent zeroth-order online optimization problem in a federated learning setting for target tracking.
The proposed solution is further analyzed in terms of errors and errors in two relevant applications.
arXiv Detail & Related papers (2023-06-09T03:51:45Z) - Compressed Regression over Adaptive Networks [58.79251288443156]
We derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem.
We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents.
arXiv Detail & Related papers (2023-04-07T13:41:08Z) - Settling the Communication Complexity for Distributed Offline
Reinforcement Learning [10.315054389907031]
We study a novel setting in offline reinforcement learning (RL) where a number of distributed machines jointly cooperate to solve the problem.
There is a budget constraint on the total number of information (in terms of bits) that each machine can send out.
For value function prediction in contextual bandits, and both episodic and non-episodic MDPs, we establish information-theoretic lower bounds on the minimax risk.
arXiv Detail & Related papers (2022-02-10T06:27:07Z) - An Information Bottleneck Approach for Controlling Conciseness in
Rationale Extraction [84.49035467829819]
We show that it is possible to better manage this trade-off by optimizing a bound on the Information Bottleneck (IB) objective.
Our fully unsupervised approach jointly learns an explainer that predicts sparse binary masks over sentences, and an end-task predictor that considers only the extracted rationale.
arXiv Detail & Related papers (2020-05-01T23:26:41Z) - Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable
Neural Distribution Alignment [52.02794488304448]
We propose a new distribution alignment method based on a log-likelihood ratio statistic and normalizing flows.
We experimentally verify that minimizing the resulting objective results in domain alignment that preserves the local structure of input domains.
arXiv Detail & Related papers (2020-03-26T22:10:04Z) - Distributed Gaussian Mean Estimation under Communication Constraints:
Optimal Rates and Communication-Efficient Algorithms [2.842794675894731]
Minimax rates of convergence, which characterize the tradeoff between the communication costs and statistical accuracy, are established.
Communication-efficient and statistically optimal procedures are developed.
arXiv Detail & Related papers (2020-01-24T04:19:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.