MDCS: More Diverse Experts with Consistency Self-distillation for Long-tailed Recognition

• URL: http://arxiv.org/abs/2308.09922v2
• Date: Thu, 30 Nov 2023 11:58:12 GMT
• Title: MDCS: More Diverse Experts with Consistency Self-distillation for Long-tailed Recognition
• Abstract summary: We propose More Diverse experts with Consistency Self-distillation (MDCS) to bridge the gap left by earlier methods. Our MDCS approach consists of two core components: Diversity Loss (DL) and Consistency Self-distillation (CS)
• Score: 14.998014782981663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, multi-expert methods have led to significant improvements in long-tail recognition (LTR). We summarize two aspects that need further enhancement to contribute to LTR boosting: (1) More diverse experts; (2) Lower model variance. However, the previous methods didn't handle them well. To this end, we propose More Diverse experts with Consistency Self-distillation (MDCS) to bridge the gap left by earlier methods. Our MDCS approach consists of two core components: Diversity Loss (DL) and Consistency Self-distillation (CS). In detail, DL promotes diversity among experts by controlling their focus on different categories. To reduce the model variance, we employ KL divergence to distill the richer knowledge of weakly augmented instances for the experts' self-distillation. In particular, we design Confident Instance Sampling (CIS) to select the correctly classified instances for CS to avoid biased/noisy knowledge. In the analysis and ablation study, we demonstrate that our method compared with previous work can effectively increase the diversity of experts, significantly reduce the variance of the model, and improve recognition accuracy. Moreover, the roles of our DL and CS are mutually reinforcing and coupled: the diversity of experts benefits from the CS, and the CS cannot achieve remarkable results without the DL. Experiments show our MDCS outperforms the state-of-the-art by 1% $\sim$ 2% on five popular long-tailed benchmarks, including CIFAR10-LT, CIFAR100-LT, ImageNet-LT, Places-LT, and iNaturalist 2018. The code is available at https://github.com/fistyee/MDCS.

Related papers

• VICAL: Vicinal Consistency Alignment for Long-Tailed Visual Recognition [32.48884030865545]
  VICAL is a framework that improves long-tailed recognition not by enforcing expert diversity, but by reducing prediction variance.<n>Our approach comprises two key components: Self-Consistency Learning and Deep Ensemble Distillation.<n>Our experiments on CIFAR-LT, ImageNet-LT, and iist 2018 show that VICAL consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-09-04T09:55:12Z)
• Rethinking Expert Training for Model Merging with Prompt Learning [29.003417422329715]
  Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model.<n>Existing approaches largely focus on improving the merging procedure itself.<n>We introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder.
  arXiv  Detail & Related papers  (2026-07-27T14:04:48Z)
• Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer [6.792743621449621]
  Multi-expert learning-to-defer surrogates are statistically consistent, yet they can underfit, suppress useful experts, or degrade as the expert pool grows.<n>We trace these failures to a shared architectural choice: casting classes and experts as actions inside one augmented prediction geometry.<n>We introduce a decoupled surrogate that estimates the class posterior with a softmax and each expert utility with an independent sigmoid.
  arXiv  Detail & Related papers  (2026-04-10T15:27:23Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Enhancing CTR Prediction with De-correlated Expert Networks [53.05653547330796]
  We propose a De-Correlated MoE (D-MoE) framework, which introduces a Cross-Expert De-Correlation loss to minimize expert correlations.<n>Extensive experiments have been conducted to validate the effectiveness of D-MoE and the de-correlation principle.
  arXiv  Detail & Related papers  (2025-05-23T14:04:38Z)
• De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts [32.1016787150064]
  Data-Free Knowledge Distillation (DFKD) is a promising task to train high-performance small models to enhance actual deployment without relying on the original training data. Existing methods commonly avoid relying on private data by utilizing synthetic or sampled data. This paper proposes a novel perspective with causal inference to disentangle the student models from the impact of such shifts.
  arXiv  Detail & Related papers  (2024-03-28T16:13:22Z)
• Mutual Distillation Learning For Person Re-Identification [27.350415735863184]
  We propose a novel approach, Mutual Distillation Learning For Person Re-identification (termed as MDPR) Our approach encompasses two branches: a hard content branch to extract local features via a uniform horizontal partitioning strategy and a Soft Content Branch to dynamically distinguish between foreground and background. Our method achieves an impressive $88.7%/94.4%$ in mAP/Rank-1 on the DukeC-reID dataset, surpassing the current state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-12T07:49:02Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [95.49699178874683]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features. We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy [84.11508381847929]
  Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks. We propose M-SMoE, which leverages routing statistics to guide expert merging. Our MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with virtually no loss in performance.
  arXiv  Detail & Related papers  (2023-10-02T16:51:32Z)
• Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation [53.94265240561697]
  We propose a novel MoE-based method called Self-Heterogeneous Integration with Knowledge Excavation (SHIKE) SHIKE achieves the state-of-the-art performance of 56.3%, 60.3%, 75.4%, and 41.9% on CIFAR100-LT (IF100), ImageNet-LT, iNaturalist 2018, and Places-LT, respectively.
  arXiv  Detail & Related papers  (2023-04-03T18:13:39Z)
• How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval [80.54532535622988]
  We show that a generalizable dense retriever can be trained to achieve high accuracy in both supervised and zero-shot retrieval. DRAGON, our dense retriever trained with diverse augmentation, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations.
  arXiv  Detail & Related papers  (2023-02-15T03:53:26Z)
• Semi-supervised Semantic Segmentation with Mutual Knowledge Distillation [20.741353967123366]
  We propose a new consistency regularization framework, termed mutual knowledge distillation (MKD) We use the pseudo-labels generated by a mean teacher to supervise the student network to achieve a mutual knowledge distillation between the two branches. Our framework outperforms previous state-of-the-art (SOTA) methods under various semi-supervised settings.
  arXiv  Detail & Related papers  (2022-08-24T12:47:58Z)
• ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot [35.309006275276374]
  One-stage long-tailed recognition methods improve the overall performance in a "seesaw" manner. Existing algorithms bypass such trade-off by a multi-stage training process. We propose a one-stage long-tailed recognition scheme, ally complementary experts (ACE)
  arXiv  Detail & Related papers  (2021-08-05T05:31:57Z)
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation [109.11580756757611]
  Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members' performances. We introduce a novel training criterion called DICE: it increases diversity by reducing spurious correlations among features.
  arXiv  Detail & Related papers  (2021-01-14T10:53:26Z)
• Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification [106.08067870620218]
  We propose a self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME) We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-06T12:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.