Collaborative Learning of Distributions under Heterogeneity and Communication Constraints

• URL: http://arxiv.org/abs/2206.00707v1
• Date: Wed, 1 Jun 2022 18:43:06 GMT
• Title: Collaborative Learning of Distributions under Heterogeneity and Communication Constraints
• Authors: Xinmeng Huang, Donghwan Lee, Edgar Dobriban, Hamed Hassani
• Abstract summary: In machine learning, users often have to collaborate to learn distributions that generate the data. We propose a novel two-stage method named SHIFT: First, the users collaborate by communicating with the server to learn a central distribution. Then, the learned central distribution is fine-tuned to estimate the individual distributions of users.
• Score: 35.82172666266493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In modern machine learning, users often have to collaborate to learn distributions that generate the data. Communication can be a significant bottleneck. Prior work has studied homogeneous users -- i.e., whose data follow the same discrete distribution -- and has provided optimal communication-efficient methods. However, these methods rely heavily on homogeneity, and are less applicable in the common case when users' discrete distributions are heterogeneous. Here we consider a natural and tractable model of heterogeneity, where users' discrete distributions only vary sparsely, on a small number of entries. We propose a novel two-stage method named SHIFT: First, the users collaborate by communicating with the server to learn a central distribution; relying on methods from robust statistics. Then, the learned central distribution is fine-tuned to estimate the individual distributions of users. We show that SHIFT is minimax optimal in our model of heterogeneity and under communication constraints. Further, we provide experimental results using both synthetic data and $n$-gram frequency estimation in the text domain, which corroborate its efficiency.

Related papers

• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional general score-mismatched diffusion samplers. We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions. This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning [50.382793324572845]
  Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. In this paper, we analyze a new method that incorporates the ideas of using data similarity and clients sampling. To address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method.
  arXiv  Detail & Related papers  (2024-09-22T00:49:10Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Probabilistic Contrastive Learning for Long-Tailed Visual Recognition [78.70453964041718]
  Longtailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. We propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T13:44:49Z)
• Federated Learning for distribution skewed data using sample weights [3.6039117546761155]
  This work focuses on improving federated learning performance for skewed data distribution across clients. The main idea is to adjust the client distribution closer to the global distribution using sample weights. We show that the proposed method not only improves federated learning accuracy but also significantly reduces communication costs.
  arXiv  Detail & Related papers  (2024-01-05T00:46:11Z)
• Probabilistic Matching of Real and Generated Data Statistics in Generative Adversarial Networks [0.6906005491572401]
  We propose a method to ensure that the distributions of certain generated data statistics coincide with the respective distributions of the real data. We evaluate the method on a synthetic dataset and a real-world dataset and demonstrate improved performance of our approach.
  arXiv  Detail & Related papers  (2023-06-19T14:03:27Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• Robust Federated Learning: The Case of Affine Distribution Shifts [41.27887358989414]
  We develop a robust federated learning algorithm that achieves satisfactory performance against distribution shifts in users' samples. We show that an affine distribution shift indeed suffices to significantly decrease the performance of the learnt classifier in a new test user.
  arXiv  Detail & Related papers  (2020-06-16T03:43:59Z)
• Synthetic Learning: Learn From Distributed Asynchronized Discriminator GAN Without Sharing Medical Image Data [21.725983290877753]
  We propose a data privacy-preserving and communication efficient distributed GAN learning framework named Distributed Asynchronized Discriminator GAN (AsynDGAN)
  arXiv  Detail & Related papers  (2020-05-29T21:05:49Z)
• Multi-Center Federated Learning [62.57229809407692]
  This paper proposes a novel multi-center aggregation mechanism for federated learning. It learns multiple global models from the non-IID user data and simultaneously derives the optimal matching between users and centers. Our experimental results on benchmark datasets show that our method outperforms several popular federated learning methods.
  arXiv  Detail & Related papers  (2020-05-03T09:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.