Ensemble Making Few-Shot Learning Stronger

• URL: http://arxiv.org/abs/2105.11904v1
• Date: Wed, 12 May 2021 17:11:10 GMT
• Title: Ensemble Making Few-Shot Learning Stronger
• Authors: Qing Lin, Yongbin Liu, Wen Wen, Zhihua Tao
• Abstract summary: This paper explores an ensemble approach to reduce the variance and introduces fine-tuning and feature attention strategies to calibrate relation-level features. Results on several few-shot relation learning tasks show that our model significantly outperforms the previous state-of-the-art models.
• Score: 4.17701749612924
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot learning has been proposed and rapidly emerging as a viable means for completing various tasks. Many few-shot models have been widely used for relation learning tasks. However, each of these models has a shortage of capturing a certain aspect of semantic features, for example, CNN on long-range dependencies part, Transformer on local features. It is difficult for a single model to adapt to various relation learning, which results in the high variance problem. Ensemble strategy could be competitive on improving the accuracy of few-shot relation extraction and mitigating high variance risks. This paper explores an ensemble approach to reduce the variance and introduces fine-tuning and feature attention strategies to calibrate relation-level features. Results on several few-shot relation learning tasks show that our model significantly outperforms the previous state-of-the-art models.

Related papers

• Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent [74.02034188307857]
  Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data. We find existing methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance. Our approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.
  arXiv  Detail & Related papers  (2025-01-02T12:45:21Z)
• RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models [60.596005921295806]
  Agglomerative models have emerged as a powerful approach to training vision foundation models. We identify critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. We propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions.
  arXiv  Detail & Related papers  (2024-12-10T17:06:41Z)
• ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization [0.32885740436059047]
  Over- parameterized neural network models often lead to significant performance discrepancies between training and test sets. We introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. We propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set.
  arXiv  Detail & Related papers  (2024-12-02T13:21:31Z)
• Bridging Domains with Approximately Shared Features [26.096779584142986]
  Multi-source domain adaptation aims to reduce performance degradation when applying machine learning models to unseen domains. Some advocate for learning invariant features from source domains, while others favor more diverse features. We propose a statistical framework that distinguishes the utilities of features based on the variance of their correlation to label $y$ across domains.
  arXiv  Detail & Related papers  (2024-03-11T04:25:41Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption [73.98706049140098]
  We propose a novel phasic content fusing few-shot diffusion model with directional distribution consistency loss. Specifically, we design a phasic training strategy with phasic content fusion to help our model learn content and style information when t is large. Finally, we propose a cross-domain structure guidance strategy that enhances structure consistency during domain adaptation.
  arXiv  Detail & Related papers  (2023-09-07T14:14:11Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)
• Byzantine Resilient Distributed Multi-Task Learning [6.850757447639822]
  We show that distributed algorithms for learning relatedness among tasks are not resilient in the presence of Byzantine agents. We propose an approach for Byzantine resilient distributed multi-task learning.
  arXiv  Detail & Related papers  (2020-10-25T04:32:52Z)
• Learning from demonstration using products of experts: applications to manipulation and task prioritization [12.378784643460474]
  We show that the fusion of models in different task spaces can be expressed as a product of experts (PoE) Multiple experiments are presented to show that learning the different models jointly in the PoE framework significantly improves the quality of the model.
  arXiv  Detail & Related papers  (2020-10-07T16:24:41Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.