Personalized Federated Learning via Variance-Aware Nonparametric Empirical Bayes

• URL: http://arxiv.org/abs/2608.09074v1
• Date: Mon, 10 Aug 2026 03:23:31 GMT
• Title: Personalized Federated Learning via Variance-Aware Nonparametric Empirical Bayes
• Abstract summary: We develop a new approach to Personalized Learning across heterogeneous clients using Nonparametric Empirical Bayes (NPEB)<n>We introduce a Variance-Aware Nonparametric Empirical Bayes (VANEB) framework that leverages the parameter-dependent performance estimation of local M-estimators.<n>While our theoretical guarantees are rooted in the regime of M-estimators, we empirically explore extensions of VANEB to modern federated learning settings involving Deep Neural Networks (DNNs)
• Score: 3.8169847142355695
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We develop a new approach to Personalized Federated Learning across heterogeneous clients using Nonparametric Empirical Bayes (NPEB). Leveraging the asymptotic normality of local parameter estimates obtained from Empirical Risk Minimization or M-estimation, our method formulates these estimates as noisy observations to estimate an unknown shared prior via Nonparametric Maximum Likelihood. A key challenge in applying NPEB in this setting is that existing approaches assume known fixed variances, which is not true in practice. To address this, we introduce a Variance-Aware Nonparametric Empirical Bayes (VANEB) framework that leverages the parameter-dependent asymptotic variance of local M-estimators. A key technical contribution is a generalized Tweedie's formula for this heteroskedastic setting. We then establish non-asymptotic error rates for density estimation in the average squared Hellinger distance and derive an oracle denoising inequality that provides error bounds for our estimator. While our theoretical guarantees are rooted in the asymptotic regime of M-estimators, we empirically explore heuristic extensions of VANEB to modern federated learning settings involving Deep Neural Networks (DNNs). For DNNs, we propose VANEB-head and VANEB-FT, which personalize the last fully connected layer via an NPEB step using an approximate diagonal variance estimator. We show that our method has strong performance on popular vision datasets MNIST and CIFAR-10, using a convolutional neural network architecture.

Related papers

• Generalized Neural Distributional Regression [0.0]
  We introduce the Generalized Neural Distributional Regression (GNDR) framework, which seamlessly embeds deep neural networks into the space of classical probability distributions.<n>We demonstrate the framework's versatility and superior distributional calibration across diverse modalities, including overdispersed clinical counts, right-ed transcriptomic survival profiles under a cure, and zero-truncated age distributions from unstructured facial images.
  arXiv  Detail & Related papers  (2026-06-16T19:01:57Z)
• Beyond Consistency: Inference for the Relative risk functional in Deep Nonparametric Cox Models [7.2275111013288535]
  It is unclear how Cox-based optimization error propagates to population risk under partial likelihood.<n>It is also unclear how pointwise bias can be controlled, and how ensemble-based uncertainty behaves under realistic variance regimes.<n>We develop an distribution theory for deep Cox estimators that addresses these issues.
  arXiv  Detail & Related papers  (2026-03-25T01:43:12Z)
• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity [49.809923981964715]
  Contaminated mixture of experts (MoE) is motivated by transfer learning methods where a pre-trained model, acting as a frozen expert, is integrated with an adapter model, functioning as a trainable expert, in order to learn a new task.<n>In this work, we characterize uniform convergence rates for estimating parameters under challenging settings where ground-truth parameters vary with the sample size.<n>We also establish corresponding minimax lower bounds to ensure that these rates are minimax optimal.
  arXiv  Detail & Related papers  (2026-01-31T23:45:50Z)
• Inference for Deep Neural Network Estimators in Generalized Nonparametric Models [8.122998173805266]
  inference on deep neural networks (DNNs) for categorical or exponential family outcomes remains underexplored.<n>We propose a DNN estimator under generalized nonparametric regression models (GNRMs) and developing a rigorous inference framework.<n>We further apply the method to the electronic Intensive Care Unit (eICU) dataset to predict ICU readmission risk and offer patient-centric insights for clinical decision making.
  arXiv  Detail & Related papers  (2025-04-12T21:32:42Z)
• Leveraging Nested MLMC for Sequential Neural Posterior Estimation with Intractable Likelihoods [0.38233569758620045]
  Methods aim to learn the posterior from adaptively proposed simulations using neural network-based conditional density estimators.<n>The automatic posterior transformation (APT) method proposed by Greenberg et al. performs well and scales to high-level runtime data.<n>In this paper, we reformulate APT as a nested estimation problem.<n>We construct several multi- Monte Carlo (MLMC) estimators for the loss function and its gradients to accommodate different scenarios.
  arXiv  Detail & Related papers  (2024-01-30T06:29:41Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Near-optimal inference in adaptive linear regression [60.08422051718195]
  Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner. We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
  arXiv  Detail & Related papers  (2021-07-05T21:05:11Z)
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization [69.07420650261649]
  We introduce a novel, simple, and powerful contrastive MI estimator named as FLO. Empirically, our FLO estimator overcomes the limitations of its predecessors and learns more efficiently. The utility of FLO is verified using an extensive set of benchmarks, which also reveals the trade-offs in practical MI estimation.
  arXiv  Detail & Related papers  (2021-07-02T15:20:41Z)
• Non-Asymptotic Performance Guarantees for Neural Estimation of $\mathsf{f}$-Divergences [22.496696555768846]
  Statistical distances quantify the dissimilarity between probability distributions. A modern method for estimating such distances from data relies on parametrizing a variational form by a neural network (NN) and optimizing it. This paper explores this tradeoff by means of non-asymptotic error bounds, focusing on three popular choices of SDs.
  arXiv  Detail & Related papers  (2021-03-11T19:47:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.