Decentralized Statistical Inference with Unrolled Graph Neural Networks
- URL: http://arxiv.org/abs/2104.01555v1
- Date: Sun, 4 Apr 2021 07:52:34 GMT
- Title: Decentralized Statistical Inference with Unrolled Graph Neural Networks
- Authors: He Wang, Yifei Shen, Ziyuan Wang, Dongsheng Li, Jun Zhang, Khaled B.
Letaief and Jie Lu
- Abstract summary: We propose a learning-based framework, which unrolls decentralized optimization algorithms into graph neural networks (GNNs)
By minimizing the recovery error via end-to-end training, this learning-based framework resolves the model mismatch issue.
Our convergence analysis reveals that the learned model parameters may accelerate the convergence and reduce the recovery error to a large extent.
- Score: 26.025935320024665
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we investigate the decentralized statistical inference
problem, where a network of agents cooperatively recover a (structured) vector
from private noisy samples without centralized coordination. Existing
optimization-based algorithms suffer from issues of model mismatch and poor
convergence speed, and thus their performance would be degraded, provided that
the number of communication rounds is limited. This motivates us to propose a
learning-based framework, which unrolls well-noted decentralized optimization
algorithms (e.g., Prox-DGD and PG-EXTRA) into graph neural networks (GNNs). By
minimizing the recovery error via end-to-end training, this learning-based
framework resolves the model mismatch issue. Our convergence analysis (with
PG-EXTRA as the base algorithm) reveals that the learned model parameters may
accelerate the convergence and reduce the recovery error to a large extent. The
simulation results demonstrate that the proposed GNN-based learning methods
prominently outperform several state-of-the-art optimization-based algorithms
in convergence speed and recovery error.
Related papers
- Convergence Visualizer of Decentralized Federated Distillation with
Reduced Communication Costs [3.2098126952615442]
Federated learning (FL) achieves collaborative learning without the need for data sharing, thus preventing privacy leakage.
This study solves two unresolved challenges of CMFD: (1) communication cost reduction and (2) visualization of model convergence.
arXiv Detail & Related papers (2023-12-19T07:23:49Z) - Achieving Linear Speedup in Decentralized Stochastic Compositional
Minimax Optimization [22.988563731766586]
The compositional minimax problem has attracted a surge of attention in recent years since it covers many emerging machine learning models.
Our study shows that the standard gossip communication strategy cannot achieve linear speedup for decentralized compositional minimax problems.
We developed a novel decentralized compositional descent ascent with momentum gradient algorithm to reduce the consensus error in the inner-level function.
arXiv Detail & Related papers (2023-07-25T11:51:20Z) - Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
We take a closer theoretical look at Independent Subnetwork Training (IST)
IST is a recently proposed and highly effective technique for solving the aforementioned problems.
We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
arXiv Detail & Related papers (2023-06-28T18:14:22Z) - Stochastic Unrolled Federated Learning [85.6993263983062]
We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning.
Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
arXiv Detail & Related papers (2023-05-24T17:26:22Z) - Network-GIANT: Fully distributed Newton-type optimization via harmonic
Hessian consensus [2.8617826964327113]
We introduce a Newton-type fully distributed optimization algorithm, Network-GIANT, based on GIANT.
We prove that our algorithm guarantees semi-global and exponential convergence to the exact solution over the network assuming strongly convex and smooth loss functions.
We provide empirical evidence of the superior convergence performance of Network-GIANT over other state-of-art distributed learning algorithms such as Network-DANE and Newton-Raphson Consensus.
arXiv Detail & Related papers (2023-05-13T11:42:40Z) - On the Convergence of Distributed Stochastic Bilevel Optimization
Algorithms over a Network [55.56019538079826]
Bilevel optimization has been applied to a wide variety of machine learning models.
Most existing algorithms restrict their single-machine setting so that they are incapable of handling distributed data.
We develop novel decentralized bilevel optimization algorithms based on a gradient tracking communication mechanism and two different gradients.
arXiv Detail & Related papers (2022-06-30T05:29:52Z) - Contextual Model Aggregation for Fast and Robust Federated Learning in
Edge Computing [88.76112371510999]
Federated learning is a prime candidate for distributed machine learning at the network edge.
Existing algorithms face issues with slow convergence and/or robustness of performance.
We propose a contextual aggregation scheme that achieves the optimal context-dependent bound on loss reduction.
arXiv Detail & Related papers (2022-03-23T21:42:31Z) - Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for
sparse recover [87.28082715343896]
We consider deep neural networks for solving inverse problems that are robust to forward model mis-specifications.
We design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem.
The proposed REST network is shown to outperform state-of-the-art model-based and data-driven algorithms in both compressive sensing and radar imaging problems.
arXiv Detail & Related papers (2021-10-20T06:15:45Z) - Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent [7.6860514640178]
We propose a novel zeroth-order optimization algorithm for distributed reinforcement learning.
It allows each agent to estimate its local gradient by cost evaluation independently, without use of any consensus protocol.
arXiv Detail & Related papers (2021-07-26T18:11:07Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.