Semantic-based Distributed Learning for Diverse and Discriminative Representations

• URL: http://arxiv.org/abs/2604.18237v1
• Date: Mon, 20 Apr 2026 13:22:58 GMT
• Title: Semantic-based Distributed Learning for Diverse and Discriminative Representations
• Abstract summary: In large-scale distributed scenarios, increasingly complex tasks demand more intelligent collaboration across networks.<n>We propose a novel distributed learning framework that ensures both diverse and discriminative representations.<n>In both cases, the resulting optimal solutions are theoretically proven to maintain discriminative and diverse properties.
• Score: 26.214742067718277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In large-scale distributed scenarios, increasingly complex tasks demand more intelligent collaboration across networks, requiring the joint extraction of structural representations from data samples. However, conventional task-specific approaches often result in nonstructural embeddings, leading to collapsed variability among data samples within the same class, particularly in classification tasks. To address this issue and fully leverage the intrinsic structure of data for downstream applications, we propose a novel distributed learning framework that ensures both diverse and discriminative representations. For independent and identically distributed (i.i.d.) data, we reformulate and decouple the global optimization function by introducing constraints on representation variance. The update rules are then derived and simplified using a primal-dual approach. For non-i.i.d. data distributions, we tackle the problem by clustering and virtually replicating nodes, allowing model updates within each cluster using block coordinate descent. In both cases, the resulting optimal solutions are theoretically proven to maintain discriminative and diverse properties, with a guaranteed convergence for i.i.d. conditions. Additionally, semantic information from representations is shared among nodes, reducing the need for common neural network architectures. Finally, extensive simulations on MNIST, CIFAR-10 and CIFAR-100 confirm the effectiveness of the proposed algorithms in capturing global structural representations.

Related papers

• Imputation Meets Clustering: Exploiting Latent Subgroup Structure for Missing Data Recovery [52.29866804948133]
  We propose CAGI (Cluster-Aware Generative Imputation), a framework that reformulates clustering and imputation as a mutually reinforcing co-optimization process.<n>Experiments on 14 benchmark datasets with 15 representative baselines demonstrate the superiority of CAGI.
  arXiv  Detail & Related papers  (2026-07-08T02:50:02Z)
• Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data [2.5157688901171995]
  Lottery Ticket Hypothesis posits that large networks contain sparseworks, or winning tickets, that can be trained in isolation to match the performance of their dense counterparts.<n>We propose the Routing Lottery (RTL), an adaptive pruning framework that discovers multiple specialized pruningworks, called adaptive tickets, each tailored to a class, cluster semantic, or environmental condition.<n>Our results recast pruning as a mechanism for aligning model structure with data heterogeneity, paving the way toward more modular and context-aware deep learning.
  arXiv  Detail & Related papers  (2026-01-29T18:56:41Z)
• Self-supervised Latent Space Optimization with Nebula Variational Coding [87.20343320266215]
  This paper proposes a variational inference model which leads to a clustered embedding.<n>We introduce additional variables in the latent space, called textbfnebula anchors, that guide the latent variables to form clusters during training.<n>Since each latent feature can be labeled with the closest anchor, we also propose to apply metric learning in a self-supervised way to make the separation between clusters more explicit.
  arXiv  Detail & Related papers  (2025-06-02T08:13:32Z)
• Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneity [40.64395367773766]
  We propose a distributed strong lottery ticket hypothesis (TH), based on which a communication-efficient personalized learning algorithm is developed.<n>In the proposed method, each local model is represented as the Hadamard product of global real-valued parameters and a personalized binary mask for pruning.<n>We provide a theoretical proof for the DSLTH, establishing it as the foundation of the proposed method.
  arXiv  Detail & Related papers  (2025-04-24T13:02:54Z)
• Client Selection in Federated Learning with Data Heterogeneity and Network Latencies [19.161254709653914]
  Federated learning (FL) is a distributed machine learning paradigm where multiple clients conduct local training based on their private data, then the updated models are sent to a central server for global aggregation.<n>In this paper, we propose two novel theoretically optimal client selection schemes that handle both these heterogeneities.
  arXiv  Detail & Related papers  (2025-04-02T17:31:15Z)
• Structural Entropy Guided Probabilistic Coding [52.01765333755793]
  We propose a novel structural entropy-guided probabilistic coding model, named SEPC.<n>We incorporate the relationship between latent variables into the optimization by proposing a structural entropy regularization loss.<n> Experimental results across 12 natural language understanding tasks, including both classification and regression tasks, demonstrate the superior performance of SEPC.
  arXiv  Detail & Related papers  (2024-12-12T00:37:53Z)
• Flexible inference in heterogeneous and attributed multilayer networks [21.349513661012498]
  We develop a probabilistic generative model to perform inference in multilayer networks with arbitrary types of information.<n>We demonstrate its ability to unveil a variety of patterns in a social support network among villagers in rural India.
  arXiv  Detail & Related papers  (2024-05-31T15:21:59Z)
• Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models [83.02797560769285]
  Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data.<n>Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts.
  arXiv  Detail & Related papers  (2024-05-26T13:11:55Z)
• AdaptDHM: Adaptive Distribution Hierarchical Model for Multi-Domain CTR Prediction [4.299153274884263]
  We propose an elegant and flexible multi-distribution modeling paradigm, named Adaptive Distribution Hierarchical Model (AdaptDHM) Our model achieves impressive prediction accuracy and its time cost during the training stage is more than 50% less than that of other models.
  arXiv  Detail & Related papers  (2022-11-22T09:10:37Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering [119.88565565454378]
  Unsupervised domain adaptation (UDA) is to learn classification models that make predictions for unlabeled data on a target domain. We propose a hybrid model of Structurally Regularized Deep Clustering, which integrates the regularized discriminative clustering of target data with a generative one. Our proposed H-SRDC outperforms all the existing methods under both the inductive and transductive settings.
  arXiv  Detail & Related papers  (2020-12-08T08:52:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.