Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study

• URL: http://arxiv.org/abs/2307.08572v4
• Date: Tue, 25 Jul 2023 10:06:18 GMT
• Title: Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study
• Authors: Luis Pedro Silvestrin, Shujian Yu, Mark Hoogendoorn
• Abstract summary: This paper revisits the robustness of the minimum error entropy criterion to deal with non-Gaussian noises. We investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common.
• Score: 16.07380451502911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coping with distributional shifts is an important part of transfer learning methods in order to perform well in real-life tasks. However, most of the existing approaches in this area either focus on an ideal scenario in which the data does not contain noises or employ a complicated training paradigm or model design to deal with distributional shifts. In this paper, we revisit the robustness of the minimum error entropy (MEE) criterion, a widely used objective in statistical signal processing to deal with non-Gaussian noises, and investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common. Specifically, we put forward a new theoretical result showing the robustness of MEE against covariate shift. We also show that by simply replacing the mean squared error (MSE) loss with the MEE on basic transfer learning algorithms such as fine-tuning and linear probing, we can achieve competitive performance with respect to state-of-the-art transfer learning algorithms. We justify our arguments on both synthetic data and 5 real-world time-series data.

Related papers

• TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression [11.040033344386366]
  We propose a two-step method with a novel fused-regularizer to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model. We extend the method to a distributed setting, allowing for a pretraining-finetuning strategy.
  arXiv  Detail & Related papers  (2024-04-01T14:58:16Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Robust Transfer Learning with Unreliable Source Data [13.276850367115333]
  We introduce a novel quantity called the ''ambiguity level'' that measures the discrepancy between the target and source regression functions. We propose a simple transfer learning procedure, and establish a general theorem that shows how this new quantity is related to the transferability of learning.
  arXiv  Detail & Related papers  (2023-10-06T21:50:21Z)
• Towards Estimating Transferability using Hard Subsets [25.86053764521497]
  We propose HASTE, a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. We show that HASTE can be used with any existing transferability metric to improve their reliability. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics.
  arXiv  Detail & Related papers  (2023-01-17T14:50:18Z)
• MaxMatch: Semi-Supervised Learning with Worst-Case Consistency [149.03760479533855]
  We propose a worst-case consistency regularization technique for semi-supervised learning (SSL) We present a generalization bound for SSL consisting of the empirical loss terms observed on labeled and unlabeled training data separately. Motivated by this bound, we derive an SSL objective that minimizes the largest inconsistency between an original unlabeled sample and its multiple augmented variants.
  arXiv  Detail & Related papers  (2022-09-26T12:04:49Z)
• The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression [26.5147705530439]
  We define a transfer learning approach to the target task as a linear regression optimization with a regularization on the distance between the to-be-learned target parameters and the already-learned source parameters. We show that for sufficiently related tasks, the optimally tuned transfer learning approach can outperform the optimally tuned ridge regression method.
  arXiv  Detail & Related papers  (2021-03-09T18:46:01Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• MMCGAN: Generative Adversarial Network with Explicit Manifold Prior [78.58159882218378]
  We propose to employ explicit manifold learning as prior to alleviate mode collapse and stabilize training of GAN. Our experiments on both the toy data and real datasets show the effectiveness of MMCGAN in alleviating mode collapse, stabilizing training, and improving the quality of generated samples.
  arXiv  Detail & Related papers  (2020-06-18T07:38:54Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels [61.8970957519509]
  This study proposes a new meta-transition-learning strategy for the task. Specifically, through the sound guidance of a small set of meta data with clean labels, the noise transition matrix and the classifier parameters can be mutually ameliorated. Our method can more accurately extract the transition matrix, naturally following its more robust performance than prior arts.
  arXiv  Detail & Related papers  (2020-06-10T07:27:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.