PrivFedTalk: Privacy-Aware Federated Diffusion with Identity-Stable Adapters for Personalized Talking-Head Generation

• URL: http://arxiv.org/abs/2604.08037v1
• Date: Thu, 09 Apr 2026 09:41:30 GMT
• Title: PrivFedTalk: Privacy-Aware Federated Diffusion with Identity-Stable Adapters for Personalized Talking-Head Generation
• Abstract summary: PrivFedTalk is a privacy-aware federated framework for personalized talking-head generation.<n>A shared diffusion backbone is trained across clients, while each client learns lightweight LoRA identity adapters from local private audio-visual data.<n>To limit update-side privacy risk, secure aggregation and client-level differential privacy are applied to adapter updates.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Talking-head generation has advanced rapidly with diffusion-based generative models, but training usually depends on centralized face-video and speech datasets, raising major privacy concerns. The problem is more acute for personalized talking-head generation, where identity-specific data are highly sensitive and often cannot be pooled across users or devices. PrivFedTalk is presented as a privacy-aware federated framework for personalized talking-head generation that combines conditional latent diffusion with parameter-efficient identity adaptation. A shared diffusion backbone is trained across clients, while each client learns lightweight LoRA identity adapters from local private audio-visual data, avoiding raw data sharing and reducing communication cost. To address heterogeneous client distributions, Identity-Stable Federated Aggregation (ISFA) weights client updates using privacy-safe scalar reliability signals computed from on-device identity consistency and temporal stability estimates. Temporal-Denoising Consistency (TDC) regularization is introduced to reduce inter-frame drift, flicker, and identity drift during federated denoising. To limit update-side privacy risk, secure aggregation and client-level differential privacy are applied to adapter updates. The implementation supports both low-memory GPU execution and multi-GPU client-parallel training on heterogeneous shared hardware. Comparative experiments on the present setup across multiple training and aggregation conditions with PrivFedTalk, FedAvg, and FedProx show stable federated optimization and successful end-to-end training and evaluation under constrained resources. The results support the feasibility of privacy-aware personalized talking-head training in federated environments, while suggesting that stronger component-wise, privacy-utility, and qualitative claims need further standardized evaluation.

Related papers

• Cooperative Local Differential Privacy: Securing Time Series Data in Distributed Environments [5.982082746731629]
  We introduce a Cooperative Local Differential Privacy (CLDP) mechanism that enhances privacy by distributing noise vectors across multiple users.<n>In our approach, noise is collaboratively generated and assigned so that when all users' data is aggregated, the noise cancels out preserving overall statistical properties while protecting individual privacy.
  arXiv  Detail & Related papers  (2025-11-12T19:52:03Z)
• DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models [17.265217612125905]
  DP-FedLoRA is a privacy-enhanced federated fine-tuning framework.<n>It integrates LoRA-based adaptation with differential privacy in a communication-efficient setting.<n>We show that DP-FedLoRA delivers competitive performance while offering strong privacy guarantees.
  arXiv  Detail & Related papers  (2025-09-11T02:16:34Z)
• CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steering [80.54309860395763]
  CoSteer is a novel collaborative framework that enables decoding-time personalization through localized delta steering.<n>We formulate token-level optimization as an online learning problem, where local delta vectors dynamically adjust the remote LLM's logits.<n>This approach preserves privacy by transmitting only the final steered tokens rather than raw data or intermediate vectors.
  arXiv  Detail & Related papers  (2025-07-07T08:32:29Z)
• Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models [12.406403248205285]
  Federated prompt personalization (FPP) is developed to address data heterogeneity and local overfitting.<n>We propose SecFPP, a secure FPP protocol harmonizing personalization, and privacy guarantees.<n>We show SecFPP significantly outperforms both non-private and privacy-preserving baselines.
  arXiv  Detail & Related papers  (2025-05-28T15:09:56Z)
• PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts [59.5243730853157]
  Large language models (LLMs) hosted on cloud servers alleviate the computational and storage burdens on local devices but raise privacy concerns.<n>Small language models (SLMs) running locally enhance privacy but suffer from limited performance on complex tasks.<n>We propose a privacy-aware wireless collaborative mixture of experts (PWC-MoE) framework to balance computational cost, performance, and privacy protection under bandwidth constraints.
  arXiv  Detail & Related papers  (2025-05-13T16:27:07Z)
• Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation [60.81109086640437]
  We propose a novel framework called Federated Retrieval-Augmented Generation (FedE4RAG)<n>FedE4RAG facilitates collaborative training of client-side RAG retrieval models.<n>We apply homomorphic encryption within federated learning to safeguard model parameters.
  arXiv  Detail & Related papers  (2025-04-27T04:26:02Z)
• Privacy-Preserving Federated Learning with Differentially Private Hyperdimensional Computing [5.667290129954206]
  We propose Federated HyperDimensional computing with Privacy-preserving (FedHDPrivacy)<n>FedHDPrivacy monitors the cumulative noise across learning rounds and adds only the additional noise required to satisfy privacy constraints.<n>In a real-world application for monitoring machining manufacturing processes, FedHDPrivacy maintains high performance while surpassing standard FL frameworks.
  arXiv  Detail & Related papers  (2024-11-02T05:00:44Z)
• Noise-Aware Algorithm for Heterogeneous Differentially Private Federated Learning [21.27813247914949]
  We propose Robust-HDP, which efficiently estimates the true noise level in clients model updates.<n>It improves utility and convergence speed, while being safe to the clients that may maliciously send falsified privacy parameter to server.
  arXiv  Detail & Related papers  (2024-06-05T17:41:42Z)
• FewFedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning [54.26614091429253]
  Federated instruction tuning (FedIT) is a promising solution, by consolidating collaborative training across multiple data owners. FedIT encounters limitations such as scarcity of instructional data and risk of exposure to training data extraction attacks. We propose FewFedPIT, designed to simultaneously enhance privacy protection and model performance of federated few-shot learning.
  arXiv  Detail & Related papers  (2024-03-10T08:41:22Z)
• Personalized Federated Learning with Attention-based Client Selection [57.71009302168411]
  We propose FedACS, a new PFL algorithm with an Attention-based Client Selection mechanism. FedACS integrates an attention mechanism to enhance collaboration among clients with similar data distributions. Experiments on CIFAR10 and FMNIST validate FedACS's superiority.
  arXiv  Detail & Related papers  (2023-12-23T03:31:46Z)
• Unsupervised Model Personalization while Preserving Privacy and Scalability: An Open Problem [55.21502268698577]
  This work investigates the task of unsupervised model personalization, adapted to continually evolving, unlabeled local user images. We provide a novel Dual User-Adaptation framework (DUA) to explore the problem. This framework flexibly disentangles user-adaptation into model personalization on the server and local data regularization on the user device.
  arXiv  Detail & Related papers  (2020-03-30T09:35:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.