A solvable model for unsupervised federated learning

• URL: http://arxiv.org/abs/2606.13045v1
• Date: Thu, 11 Jun 2026 08:30:40 GMT
• Title: A solvable model for unsupervised federated learning
• Abstract summary: We introduce a theoretical framework for analyzing federated learning in a generative setting through a teacher-multiple interacting students scenario.<n>We analytically show that interactions among students systematically enhance learning performance.
• Score: 7.720611301237503
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We introduce a theoretical framework for analyzing federated learning in a generative setting through a teacher-multiple interacting students scenario, in which each student receives a distinct realization of the data, either through a different noise corruption or by accessing a different subset, possibly of varying size. Using theoretical tools in equilibrium disordered system, we analytically show that interactions among students systematically enhance learning performance: highly noisy students require fewer samples to recover the underlying pattern, while low-noise students achieve a larger overlap with the ground-truth signal. We derive the optimal Bayesian conditions for teacher recovery as functions of the sample complexity, noise level, and interaction strength, and validate these predictions through numerical simulations. The resulting dynamics can be mapped onto equilibrium sampling in a Restricted Boltzmann Machine with a structured hidden layer, providing a principled theoretical understanding of how interactions improve distributed generative modeling.

Related papers

• Theoretical Analysis of Contrastive Learning under Imbalanced Data: From Training Dynamics to a Pruning Solution [33.0181633510156]
  We develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data.<n>Our results reveal that neuron weights evolve through three distinct stages of training, with different dynamics for majority features, minority features, and noise.<n>Inspired by these neuron-level behaviors, we show that pruning restores performance degraded by imbalance and enhances feature separation, offering both conceptual insights and practical guidance.
  arXiv  Detail & Related papers  (2026-02-10T23:06:12Z)
• FANoise: Singular Value-Adaptive Noise Modulation for Robust Multimodal Representation Learning [24.94576263410761]
  We study the role of noise gradient in representation learning from both-based and feature distribution perspectives.<n>We propose FANoise, a novel feature-adaptive noise injection strategy.<n>Under this framework, experiments demonstrate that FANoise consistently improves overall performance on multimodal tasks.
  arXiv  Detail & Related papers  (2025-11-26T02:50:29Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Regularized Neural Ensemblers [55.15643209328513]
  In this study, we explore employing regularized neural networks as ensemble methods.<n>Motivated by the risk of learning low-diversity ensembles, we propose regularizing the ensembling model by randomly dropping base model predictions.<n>We demonstrate this approach provides lower bounds for the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective [60.64922606733441]
  We introduce a mathematical model that formalizes relational learning as hypergraph recovery to study pre-training of Foundation Models (FMs) In our framework, the world is represented as a hypergraph, with data abstracted as random samples from hyperedges. We theoretically examine the feasibility of a Pre-Trained Model (PTM) to recover this hypergraph and analyze the data efficiency in a minimax near-optimal style.
  arXiv  Detail & Related papers  (2024-06-17T06:20:39Z)
• A Mathematical Model of the Hidden Feedback Loop Effect in Machine Learning Systems [44.99833362998488]
  We introduce a repeated learning process to jointly describe several phenomena attributed to unintended hidden feedback loops. A distinctive feature of such repeated learning setting is that the state of the environment becomes causally dependent on the learner itself over time. We present a novel dynamical systems model of the repeated learning process and prove the limiting set of probability distributions for positive and negative feedback loop modes.
  arXiv  Detail & Related papers  (2024-05-04T17:57:24Z)
• The Copycat Perceptron: Smashing Barriers Through Collective Learning [3.55026004901472]
  We analyze a general setting in which thermal noise is present that affects each student's generalization performance. We find that the coupling of replicas leads to a bend of the phase diagram towards smaller values of $alpha$. These results provide additional analytic and numerical evidence for the recently conjectured Bayes-optimal property of Replicated Simulated Annealing.
  arXiv  Detail & Related papers  (2023-08-07T17:51:09Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Understanding Robustness in Teacher-Student Setting: A New Perspective [42.746182547068265]
  Adrial examples are machine learning models where bounded adversarial perturbation could mislead the models to make arbitrarily incorrect predictions. Extensive studies try to explain the existence of adversarial examples and provide ways to improve model robustness. Our studies could shed light on the future exploration about adversarial examples, and enhancing model robustness via principled data augmentation.
  arXiv  Detail & Related papers  (2021-02-25T20:54:24Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.