PIcsC: Partitioning-Induced Covariate Shift Correction

• URL: http://arxiv.org/abs/2607.25441v1
• Date: Tue, 28 Jul 2026 08:33:04 GMT
• Title: PIcsC: Partitioning-Induced Covariate Shift Correction
• Abstract summary: We propose a Fisher information-based regularization framework that mitigates distribution mismatch between data partitions and a reference distribution.<n>textttPIcsC approximates partition divergence using the Fisher Information Matrix (FIM) and incorporates the resulting statistic as a regularizer during optimization.
• Score: 4.266675330597196
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Covariate shift across training-data partitions biases model selection and parameter estimation in cross-validation, lifelong learning, and federated learning. We propose \textit{Partition-Induced Covariate-shift Correction} (\texttt{PIcsC}), a Fisher information-based regularization framework that mitigates distribution mismatch between data partitions and a reference distribution. \texttt{PIcsC} approximates partition divergence using the Fisher Information Matrix (FIM) and incorporates the resulting statistic as a regularizer during optimization. The same formulation applies to both centrally partitioned datasets (batches or cross-validation folds) and inherently distributed data (federated clients or decentralized nodes), requiring only partition-local gradient statistics rather than raw data. We further introduce a conditional adaptation mechanism that combines FIM shift with KL divergence to detect significant distribution shifts and activates regularization only when necessary. Experiments on more than 40 datasets demonstrate consistent improvements under both natural and synthetic covariate shift. On fragmented batch and fold settings, \texttt{PIcsC} reduces fragmentation-induced performance degradation by more than 20\% and 25\%, respectively. On seven federated learning benchmarks, it consistently outperforms FedAvg, FedProx, and SCAFFOLD by 3 -5 percentage points without requiring client-specific personalization. These results demonstrate that Fisher information provides an effective and unified mechanism for mitigating partition-induced covariate shift across both centralized and distributed learning.

Related papers

• Entropy-Regularized Probabilistic Gates for Sparse Model Discovery in Scarce-Data Federated Learning [0.0]
  Federated learning (FL) is a distributed machine learning (ML) paradigm with collaboration among multiple clients without sharing data.<n>We study entropy regularization of gate distributions as a mechanism to maintain uncertainty in sparse federated optimization.
  arXiv  Detail & Related papers  (2026-06-30T23:51:44Z)
• BESplit: Bias-Compensated Split Federated Learning with Evidential Aggregation [2.5015678499211407]
  Split Federated Learning (SFL) enables privacy-preserving collaborative training by partitioning models between clients and a server.<n>Under non-IID data distributions, SFL often suffers from biased optimization and unstable convergence.<n>We propose BESplit, an architecture-aware framework that exploits the intrinsic structure of SFL to mitigate non-IID effects.
  arXiv  Detail & Related papers  (2026-05-17T15:42:33Z)
• Technical note on Fisher Information for Robust Federated Cross-Validation [3.5808917363708743]
  We propose Fisher Information for Robust fEderated validation (textbfFIRE)<n>Fire outperforms importance weighting benchmarks by $5.1%$ at maximum and federated learning benchmarks by up to $5.3%$ on shifted validation sets.
  arXiv  Detail & Related papers  (2025-10-04T15:30:04Z)
• Closed-Form Beta Distribution Estimation from Sparse Statistics with Random Forest Implicit Regularization [0.0]
  This work advances distribution recovery from sparse data and ensemble classification through three main contributions.<n>First, we introduce a closed-form estimator that reconstructs scaled beta distributions from limited statistics.<n>Second, we establish a link between classification accuracy and distributional closeness by deriving error bounds.<n>Third, we show that zero-variance features act as an implicit regularizer, increasing selection probability for mid-ranked predictors.
  arXiv  Detail & Related papers  (2025-07-31T17:55:07Z)
• Interaction-Aware Gaussian Weighting for Clustered Federated Learning [58.92159838586751]
  Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy.<n>We propose a novel clustered FL method, FedGWC (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution.<n>Our experiments on benchmark datasets show that FedGWC outperforms existing FL algorithms in cluster quality and classification accuracy.
  arXiv  Detail & Related papers  (2025-02-05T16:33:36Z)
• Mitigating covariate shift in non-colocated data with learned parameter priors [0.0]
  We present textitFragmentation-induced co-shift remediation ($FIcsR$), which minimizes an $f$-divergence between a fragment's covariate distribution and that of the standard cross-validation baseline. We run extensive classification experiments on multiple data classes, over $40$ datasets, and with data batched over multiple sequence lengths. The results are promising under all these conditions; with improved accuracy against batch and fold state-of-the-art by more than $5%$ and $10%$, respectively.
  arXiv  Detail & Related papers  (2024-11-10T15:48:29Z)
• Improving Distribution Alignment with Diversity-based Sampling [0.0]
  Domain shifts are ubiquitous in machine learning, and can substantially degrade a model's performance when deployed to real-world data. This paper proposes to improve these estimates by inducing diversity in each sampled minibatch. It simultaneously balances the data and reduces the variance of the gradients, thereby enhancing the model's generalisation ability.
  arXiv  Detail & Related papers  (2024-10-05T17:26:03Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Learnable Distribution Calibration for Few-Shot Class-Incremental Learning [122.2241120474278]
  Few-shot class-incremental learning (FSCIL) faces challenges of memorizing old class distributions and estimating new class distributions given few training samples. We propose a learnable distribution calibration (LDC) approach, with the aim to systematically solve these two challenges using a unified framework.
  arXiv  Detail & Related papers  (2022-10-01T09:40:26Z)
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction [48.85303253333453]
  Federated learning (FL) allows multiple clients to collectively train a high-performance global model without sharing their private data. We propose a novel federated learning algorithm with local drift decoupling and correction (FedDC) Our FedDC only introduces lightweight modifications in the local training phase, in which each client utilizes an auxiliary local drift variable to track the gap between the local model parameter and the global model parameters. Experiment results and analysis demonstrate that FedDC yields expediting convergence and better performance on various image classification tasks.
  arXiv  Detail & Related papers  (2022-03-22T14:06:26Z)
• Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization [89.73665256847858]
  We show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet. We also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS.
  arXiv  Detail & Related papers  (2021-07-09T19:48:23Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• Robust Federated Learning: The Case of Affine Distribution Shifts [41.27887358989414]
  We develop a robust federated learning algorithm that achieves satisfactory performance against distribution shifts in users' samples. We show that an affine distribution shift indeed suffices to significantly decrease the performance of the learnt classifier in a new test user.
  arXiv  Detail & Related papers  (2020-06-16T03:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.