Improving out-of-distribution generalization via multi-task self-supervised pretraining

• URL: http://arxiv.org/abs/2003.13525v1
• Date: Mon, 30 Mar 2020 14:55:53 GMT
• Title: Improving out-of-distribution generalization via multi-task self-supervised pretraining
• Authors: Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Keskar, and Richard Socher
• Abstract summary: We show that features obtained using self-supervised learning are comparable to, or better than, supervised learning for domain generalization in computer vision. We introduce a new self-supervised pretext task of predicting responses to Gabor filter banks.
• Score: 48.29123326140466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised feature representations have been shown to be useful for supervised classification, few-shot learning, and adversarial robustness. We show that features obtained using self-supervised learning are comparable to, or better than, supervised learning for domain generalization in computer vision. We introduce a new self-supervised pretext task of predicting responses to Gabor filter banks and demonstrate that multi-task learning of compatible pretext tasks improves domain generalization performance as compared to training individual tasks alone. Features learnt through self-supervision obtain better generalization to unseen domains when compared to their supervised counterpart when there is a larger domain shift between training and test distributions and even show better localization ability for objects of interest. Self-supervised feature representations can also be combined with other domain generalization methods to further boost performance.

Related papers

• Uniting contrastive and generative learning for event sequences models [51.547576949425604]
  This study investigates the integration of two self-supervised learning techniques - instance-wise contrastive learning and a generative approach based on restoring masked events in latent space. Experiments conducted on several public datasets, focusing on sequence classification and next-event type prediction, show that the integrated method achieves superior performance compared to individual approaches.
  arXiv  Detail & Related papers  (2024-08-19T13:47:17Z)
• Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning [10.784911682565879]
  Ensembling a neural network is a widely recognized approach to enhance model performance, estimate uncertainty, and improve robustness in deep supervised learning. We present a novel self-supervised training regime that leverages an ensemble of independent sub-networks. Our method efficiently builds a sub-model ensemble with high diversity, leading to well-calibrated estimates of model uncertainty.
  arXiv  Detail & Related papers  (2023-08-28T16:58:44Z)
• Semi-supervised learning made simple with self-supervised clustering [65.98152950607707]
  Self-supervised learning models have been shown to learn rich visual representations without requiring human annotations. We propose a conceptually simple yet empirically powerful approach to turn clustering-based self-supervised methods into semi-supervised learners.
  arXiv  Detail & Related papers  (2023-06-13T01:09:18Z)
• Functional Knowledge Transfer with Self-supervised Representation Learning [11.566644244783305]
  This work investigates the unexplored usability of self-supervised representation learning in the direction of functional knowledge transfer. In this work, functional knowledge transfer is achieved by joint optimization of self-supervised learning pseudo task and supervised learning task.
  arXiv  Detail & Related papers  (2023-03-12T21:14:59Z)
• Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
  Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task. We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
  arXiv  Detail & Related papers  (2022-10-13T17:59:16Z)
• Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularization [12.892376738542383]
  We propose an effective self-supervised learning framework and a novel regularization strategy to facilitate self-supervised speaker representation learning. With our strong online data augmentation strategy, the proposed SSReg shows the potential of self-supervised learning without using negative pairs. Comprehensive experiments on the VoxCeleb datasets demonstrate that our proposed self-supervised approach obtains a 23.4% relative improvement.
  arXiv  Detail & Related papers  (2021-12-08T18:41:19Z)
• How Well Do Self-Supervised Models Transfer? [92.16372657233394]
  We evaluate the transfer performance of 13 top self-supervised models on 40 downstream tasks. We find ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition. No single self-supervised method dominates overall, suggesting that universal pre-training is still unsolved.
  arXiv  Detail & Related papers  (2020-11-26T16:38:39Z)
• Self-Supervised Learning Across Domains [33.86614301708017]
  We propose to apply a similar approach to the problem of object recognition across domains. Our model learns the semantic labels in a supervised fashion, and broadens its understanding of the data by learning from self-supervised signals on the same images. This secondary task helps the network to focus on object shapes, learning concepts like spatial orientation and part correlation, while acting as a regularizer for the classification task.
  arXiv  Detail & Related papers  (2020-07-24T06:19:53Z)
• Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization [95.73898853032865]
  We present a new domain generalization framework that learns how to generalize across domains simultaneously. We demonstrate the effectiveness of our approach on two standard object recognition benchmarks.
  arXiv  Detail & Related papers  (2020-07-18T03:12:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.