Measuring Heterogeneity in Machine Learning with Distributed Energy Distance

• URL: http://arxiv.org/abs/2501.16174v1
• Date: Mon, 27 Jan 2025 16:15:57 GMT
• Title: Measuring Heterogeneity in Machine Learning with Distributed Energy Distance
• Authors: Mengchen Fan, Baocheng Geng, Roman Shterenberg, Joseph A. Casey, Zhong Chen, Keren Li,
• Abstract summary: We introduce energy distance as a sensitive measure for quantifying distributional discrepancies. We develop Taylor approximations that preserve key theoretical quantitative properties while reducing computational overhead. We propose a novel application of energy distance to assign penalty weights for aligning predictions across heterogeneous nodes.
• Score: 3.8318398579197335
• License:
• Abstract: In distributed and federated learning, heterogeneity across data sources remains a major obstacle to effective model aggregation and convergence. We focus on feature heterogeneity and introduce energy distance as a sensitive measure for quantifying distributional discrepancies. While we show that energy distance is robust for detecting data distribution shifts, its direct use in large-scale systems can be prohibitively expensive. To address this, we develop Taylor approximations that preserve key theoretical quantitative properties while reducing computational overhead. Through simulation studies, we show how accurately capturing feature discrepancies boosts convergence in distributed learning. Finally, we propose a novel application of energy distance to assign penalty weights for aligning predictions across heterogeneous nodes, ultimately enhancing coordination in federated and distributed settings.

Related papers

• Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shift [9.530897053573186]
  This paper introduces a novel dissimilarity measure that utilizes vicinity information, i.e., the local structure of data points. We characterize the excess error using the proposed measure and demonstrate faster or competitive convergence rates compared to previous techniques.
  arXiv  Detail & Related papers  (2024-05-27T07:55:27Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Distributed Markov Chain Monte Carlo Sampling based on the Alternating Direction Method of Multipliers [143.6249073384419]
  In this paper, we propose a distributed sampling scheme based on the alternating direction method of multipliers. We provide both theoretical guarantees of our algorithm's convergence and experimental evidence of its superiority to the state-of-the-art. In simulation, we deploy our algorithm on linear and logistic regression tasks and illustrate its fast convergence compared to existing gradient-based methods.
  arXiv  Detail & Related papers  (2024-01-29T02:08:40Z)
• Distributed Variational Inference for Online Supervised Learning [15.038649101409804]
  This paper develops a scalable distributed probabilistic inference algorithm. It applies to continuous variables, intractable posteriors and large-scale real-time data in sensor networks.
  arXiv  Detail & Related papers  (2023-09-05T22:33:02Z)
• Data-heterogeneity-aware Mixing for Decentralized Learning [63.83913592085953]
  We characterize the dependence of convergence on the relationship between the mixing weights of the graph and the data heterogeneity across nodes. We propose a metric that quantifies the ability of a graph to mix the current gradients. Motivated by our analysis, we propose an approach that periodically and efficiently optimize the metric.
  arXiv  Detail & Related papers  (2022-04-13T15:54:35Z)
• Cycle Consistent Probability Divergences Across Different Spaces [38.43511529063335]
  Discrepancy measures between probability distributions are at the core of statistical inference and machine learning. This work proposes a novel unbalanced Monge optimal transport formulation for matching, up to isometries, distributions on different spaces.
  arXiv  Detail & Related papers  (2021-11-22T16:35:58Z)
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty [58.144520501201995]
  Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
  arXiv  Detail & Related papers  (2021-10-12T22:04:19Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• Targeted free energy estimation via learned mappings [66.20146549150475]
  Free energy perturbation (FEP) was proposed by Zwanzig more than six decades ago as a method to estimate free energy differences. FEP suffers from a severe limitation: the requirement of sufficient overlap between distributions. One strategy to mitigate this problem, called Targeted Free Energy Perturbation, uses a high-dimensional mapping in configuration space to increase overlap.
  arXiv  Detail & Related papers  (2020-02-12T11:10:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.