An Extensive Study on D2C: Overfitting Remediation in Deep Learning Using a Decentralized Approach

• URL: http://arxiv.org/abs/2411.15876v1
• Date: Sun, 24 Nov 2024 15:31:22 GMT
• Title: An Extensive Study on D2C: Overfitting Remediation in Deep Learning Using a Decentralized Approach
• Authors: Md. Saiful Bari Siddiqui, Md Mohaiminul Islam, Md. Golam Rabiul Alam,
• Abstract summary: Divide2Conquer (D2C) is a technique to mitigate overfitting in deep learning. D2C partitions the training data into multiple subsets and trains identical models independently on each subset. Empirical evaluations on benchmark datasets demonstrate that D2C significantly enhances generalization performance.
• Score: 2.7275555568872702
• License:
• Abstract: Overfitting remains a significant challenge in deep learning, often arising from data outliers, noise, and limited training data. To address this, we propose Divide2Conquer (D2C), a novel technique to mitigate overfitting. D2C partitions the training data into multiple subsets and trains identical models independently on each subset. To balance model generalization and subset-specific learning, the model parameters are periodically aggregated and averaged during training. This process enables the learning of robust patterns while minimizing the influence of outliers and noise. Empirical evaluations on benchmark datasets across diverse deep-learning tasks demonstrate that D2C significantly enhances generalization performance, particularly with larger datasets. Our analysis includes evaluations of decision boundaries, loss curves, and other performance metrics, highlighting D2C's effectiveness both as a standalone technique and in combination with other overfitting reduction methods. We further provide a rigorous mathematical justification for D2C's underlying principles and examine its applicability across multiple domains. Finally, we explore the trade-offs associated with D2C and propose strategies to address them, offering a holistic view of its strengths and limitations. This study establishes D2C as a versatile and effective approach to combating overfitting in deep learning. Our codes are publicly available at: https://github.com/Saiful185/Divide2Conquer.

Related papers

• DANCE: Dual-View Distribution Alignment for Dataset Condensation [39.08022095906364]
  We propose a new DM-based method named Dual-view distribution AligNment for dataset CondEnsation (DANCE) Specifically, from the inner-class view, we construct multiple "middle encoders" to perform pseudo long-term distribution alignment. While from the inter-class view, we use the expert models to perform distribution calibration.
  arXiv  Detail & Related papers  (2024-06-03T07:22:17Z)
• Learning to Maximize Mutual Information for Chain-of-Thought Distillation [13.660167848386806]
  Distilling Step-by-Step(DSS) has demonstrated promise by imbuing smaller models with the superior reasoning capabilities of their larger counterparts. However, DSS overlooks the intrinsic relationship between the two training tasks, leading to ineffective integration of CoT knowledge with the task of label prediction. We propose a variational approach to solve this problem using a learning-based method.
  arXiv  Detail & Related papers  (2024-03-05T22:21:45Z)
• D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning [70.98091101459421]
  Coreset selection seeks to select a subset of the training data so as to maximize the performance of models trained on this subset, also referred to as coreset. We propose a novel pruning algorithm, D2 Pruning, that uses forward and reverse message passing over this dataset graph for coreset selection. Results show that D2 Pruning improves coreset selection over previous state-of-the-art methods for up to 70% pruning rates.
  arXiv  Detail & Related papers  (2023-10-11T23:01:29Z)
• Towards Sustainable Learning: Coresets for Data-efficient Deep Learning [9.51481812606879]
  CREST is the first scalable subsetx deep networks framework with rigorous theoretical subsetx experiments on datasets. CREST identifies the most valuable examples of a non-Image function.
  arXiv  Detail & Related papers  (2023-06-02T02:51:08Z)
• Tackling Long-Tailed Category Distribution Under Domain Shifts [50.21255304847395]
  Existing approaches cannot handle the scenario where both issues exist. We designed three novel core functional blocks including Distribution Calibrated Classification Loss, Visual-Semantic Mapping and Semantic-Similarity Guided Augmentation. Two new datasets were proposed for this problem, named AWA2-LTS and ImageNet-LTS.
  arXiv  Detail & Related papers  (2022-07-20T19:07:46Z)
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition [88.34182299496074]
  Action labels are only available on a source dataset, but unavailable on a target dataset in the training stage. We utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets. By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks.
  arXiv  Detail & Related papers  (2022-07-17T07:05:39Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)
• Dynamic Network-Assisted D2D-Aided Coded Distributed Learning [59.29409589861241]
  We propose a novel device-to-device (D2D)-aided coded federated learning method (D2D-CFL) for load balancing across devices. We derive an optimal compression rate for achieving minimum processing time and establish its connection with the convergence time. Our proposed method is beneficial for real-time collaborative applications, where the users continuously generate training data.
  arXiv  Detail & Related papers  (2021-11-26T18:44:59Z)
• Learning Robust Representation for Clustering through Locality Preserving Variational Discriminative Network [16.259673823482665]
  Variational Deep Embedding achieves great success in various clustering tasks. VaDE suffers from two problems: 1) it is fragile to the input noise; 2) it ignores the locality information between the neighboring data points. We propose a joint learning framework that improves VaDE with a robust embedding discriminator and a local structure constraint.
  arXiv  Detail & Related papers  (2020-12-25T02:31:55Z)
• One-Shot Object Detection without Fine-Tuning [62.39210447209698]
  We introduce a two-stage model consisting of a first stage Matching-FCOS network and a second stage Structure-Aware Relation Module. We also propose novel training strategies that effectively improve detection performance. Our method exceeds the state-of-the-art one-shot performance consistently on multiple datasets.
  arXiv  Detail & Related papers  (2020-05-08T01:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.