Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models

• URL: http://arxiv.org/abs/2603.12977v2
• Date: Mon, 16 Mar 2026 09:17:09 GMT
• Title: Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models
• Abstract summary: Foundation models are commonly deployed as frozen feature extractors with a small trainable head to adapt to private, user-generated data in federated settings.<n>We study this problem in a practically relevant but under-explored regime: a frozen foundation model with a ridge-regression head.
• Score: 9.98952145483793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models are commonly deployed as frozen feature extractors with a small trainable head to adapt to private, user-generated data in federated settings. The ``right to be forgotten'' requires removing the influence of specific samples or users from the trained model on demand. Existing federated unlearning methods target general deep models and rely on approximate reconstruction or selective retraining, making exactness costly or elusive. We study this problem in a practically relevant but under-explored regime: a frozen foundation model with a ridge-regression head. The exact optimum depends on the data only through two additive sufficient statistics, which we turn into a communication protocol supporting an arbitrary stream of add and delete requests via fixed-size messages. The server maintains a head that is, in exact arithmetic, pointwise identical to centralized retraining after every request. We provide deterministic retrain-equivalence guarantees, order and partition invariance, two server-side variants, and a Bayesian certificate of zero KL divergence. Experiments on four benchmarks confirm the guarantees: both variants match centralized ridge retraining to within $10^{-9}$ relative Frobenius error and complete each request at orders-of-magnitude lower cost than federated retraining baselines.

Related papers

• Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference [5.67775962599719]
  Hybrid fully homomorphic encryption(FHE) inference improves the practicality of private inference.<n>Recent schemes attempt to protect model confidentiality by returning noisy, output-permuted responses.<n>We show that this protection fails in the correctness regime required by hybrid FHE systems.
  arXiv  Detail & Related papers  (2026-09-11T14:37:41Z)
• Simplex Relaxation for Discrete Diffusion [58.33696160279345]
  We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable.<n>Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation.
  arXiv  Detail & Related papers  (2026-08-11T07:58:17Z)
• DECAF: De-Clustering for Adaptive Representational Unlearning [13.298390892944221]
  We propose DECAF (DE-Clustering for Adaptive Forgetting), a post-hoc method that operates only on the forget set and is designed to break the cluster.<n>On CIFAR-10 with ResNet-18, DECAF attains 0.10% forget-class accuracy, 79.4% retain accuracy, and an AUS of 0.88, surpassing all other baselines.
  arXiv  Detail & Related papers  (2026-07-27T02:11:58Z)
• One-Shot Federated Ridge Regression: Exact Recovery via Sufficient Statistic Aggregation [0.7106986689736825]
  Federated ridge regression is a distributed equilibrium problem where each client computes local sufficient statistics and transmits them once.<n>We establish differential privacy guarantees where noise is injected once per client, eliminating the composition penalty that degrades privacy in multi-round protocols.<n>Experiments on synthetic heterogeneous regression demonstrate that one-shot fusion matches FedAvg accuracy while requiring up to $38times$ less communication.
  arXiv  Detail & Related papers  (2026-01-13T04:47:22Z)
• Fully Decentralized Certified Unlearning [4.944495309580904]
  Machine unlearning (MUNIST) seeks to remove the influence of specified data from a trained model in response to privacy requests or data poisoning.<n>While certified unlearning has been analyzed in centralized and federated settings (via guarantees of differential privacy, DP), decentralized setting -- peers communicate without a coordinator underexplored.
  arXiv  Detail & Related papers  (2025-12-09T10:15:15Z)
• Tackling Federated Unlearning as a Parameter Estimation Problem [2.9085589574462816]
  This work introduces an efficient Federated Unlearning framework based on information theory.<n>Our method uses second-order Hessian information to identify and selectively reset only the parameters most sensitive to the data being forgotten.
  arXiv  Detail & Related papers  (2025-08-26T14:24:45Z)
• Distributed Training under Packet Loss [8.613477072763404]
  Leveraging unreliable connections will reduce latency but may sacrifice model accuracy and convergence once packets are dropped.<n>We introduce a principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss.<n>This work bridges the gap between communication-efficient protocols and the accuracy and guarantees demanded by modern large-model training.
  arXiv  Detail & Related papers  (2025-07-02T11:07:20Z)
• SPIRE: Conditional Personalization for Federated Diffusion Generative Models [7.8583640700306585]
  Shared Backbone Personal Identity Representation Embeddings (SPIRE) is a framework that casts per client diffusion based generation as conditional generation in FL.<n>SPIRE factorizes the network into (i) a high capacity global backbone that learns a population level score function and (ii) lightweight, learnable client embeddings that encode local data statistics.<n>Our analysis also hints at how client embeddings act as biases that steer a shared score network toward personalized distributions.
  arXiv  Detail & Related papers  (2025-06-14T01:40:31Z)
• Test-Time Alignment via Hypothesis Reweighting [56.71167047381817]
  Large pretrained models often struggle with underspecified tasks.<n>We propose a novel framework to address the challenge of aligning models to test-time user intent.
  arXiv  Detail & Related papers  (2024-12-11T23:02:26Z)
• Accelerating Heterogeneous Federated Learning with Closed-form Classifiers [23.133964735844007]
  Federated Learning (FL) methods often struggle in highly statistically heterogeneous settings. We introduce Federated Recursive Ridge Regression (Fed3R) Fed3R is immune to statistical heterogeneity and is invariant to the sampling order of the clients.
  arXiv  Detail & Related papers  (2024-06-03T08:52:06Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Variational Bayesian Unlearning [54.26984662139516]
  We study the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased. We show that it is equivalent to minimizing an evidence upper bound which trades off between fully unlearning from erased data vs. not entirely forgetting the posterior belief. In model training with VI, only an approximate (instead of exact) posterior belief given the full data can be obtained, which makes unlearning even more challenging.
  arXiv  Detail & Related papers  (2020-10-24T11:53:00Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)
• The Right Tool for the Job: Matching Model and Instance Complexities [62.95183777679024]
  As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. We propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks.
  arXiv  Detail & Related papers  (2020-04-16T04:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.