Task-Agnostic Robust Representation Learning
- URL: http://arxiv.org/abs/2203.07596v1
- Date: Tue, 15 Mar 2022 02:05:11 GMT
- Title: Task-Agnostic Robust Representation Learning
- Authors: A. Tuan Nguyen, Ser Nam Lim, Philip Torr
- Abstract summary: We study the problem of robust representation learning with unlabeled data in a task-agnostic manner.
We derive an upper bound on the adversarial loss of a prediction model on any downstream task, using its loss on the clean data and a robustness regularizer.
Our method achieves preferable adversarial performance compared to relevant baselines.
- Score: 31.818269301504564
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: It has been reported that deep learning models are extremely vulnerable to
small but intentionally chosen perturbations of its input. In particular, a
deep network, despite its near-optimal accuracy on the clean images, often
mis-classifies an image with a worst-case but humanly imperceptible
perturbation (so-called adversarial examples). To tackle this problem, a great
amount of research has been done to study the training procedure of a network
to improve its robustness. However, most of the research so far has focused on
the case of supervised learning. With the increasing popularity of
self-supervised learning methods, it is also important to study and improve the
robustness of their resulting representation on the downstream tasks. In this
paper, we study the problem of robust representation learning with unlabeled
data in a task-agnostic manner. Specifically, we first derive an upper bound on
the adversarial loss of a prediction model (which is based on the learned
representation) on any downstream task, using its loss on the clean data and a
robustness regularizer. Moreover, the regularizer is task-independent, thus we
propose to minimize it directly during the representation learning phase to
make the downstream prediction model more robust. Extensive experiments show
that our method achieves preferable adversarial performance compared to
relevant baselines.
Related papers
- The Trade-off between Universality and Label Efficiency of
Representations from Contrastive Learning [32.15608637930748]
We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously.
We provide analysis using a theoretical data model and show that, while more diverse pre-training data result in more diverse features for different tasks, it puts less emphasis on task-specific features.
arXiv Detail & Related papers (2023-02-28T22:14:33Z) - A Comprehensive Study on Robustness of Image Classification Models:
Benchmarking and Rethinking [54.89987482509155]
robustness of deep neural networks is usually lacking under adversarial examples, common corruptions, and distribution shifts.
We establish a comprehensive benchmark robustness called textbfARES-Bench on the image classification task.
By designing the training settings accordingly, we achieve the new state-of-the-art adversarial robustness.
arXiv Detail & Related papers (2023-02-28T04:26:20Z) - Towards Robust Dataset Learning [90.2590325441068]
We propose a principled, tri-level optimization to formulate the robust dataset learning problem.
Under an abstraction model that characterizes robust vs. non-robust features, the proposed method provably learns a robust dataset.
arXiv Detail & Related papers (2022-11-19T17:06:10Z) - Probing Representation Forgetting in Supervised and Unsupervised
Continual Learning [14.462797749666992]
Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model.
We show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning.
arXiv Detail & Related papers (2022-03-24T23:06:08Z) - What Makes Good Contrastive Learning on Small-Scale Wearable-based
Tasks? [59.51457877578138]
We study contrastive learning on the wearable-based activity recognition task.
This paper presents an open-source PyTorch library textttCL-HAR, which can serve as a practical tool for researchers.
arXiv Detail & Related papers (2022-02-12T06:10:15Z) - Self-Damaging Contrastive Learning [92.34124578823977]
Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution.
This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes.
Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
arXiv Detail & Related papers (2021-06-06T00:04:49Z) - An Effective Baseline for Robustness to Distributional Shift [5.627346969563955]
Refraining from confidently predicting when faced with categories of inputs different from those seen during training is an important requirement for the safe deployment of deep learning systems.
We present a simple, but highly effective approach to deal with out-of-distribution detection that uses the principle of abstention.
arXiv Detail & Related papers (2021-05-15T00:46:11Z) - Low-Regret Active learning [64.36270166907788]
We develop an online learning algorithm for identifying unlabeled data points that are most informative for training.
At the core of our work is an efficient algorithm for sleeping experts that is tailored to achieve low regret on predictable (easy) instances.
arXiv Detail & Related papers (2021-04-06T22:53:45Z) - Enabling the Network to Surf the Internet [13.26679087834881]
We develop a framework that enables the model to surf the Internet.
We observe that the generalization ability of the learned representation is crucial for self-supervised learning.
We demonstrate the superiority of the proposed framework with experiments on miniImageNet, tieredImageNet and Omniglot.
arXiv Detail & Related papers (2021-02-24T11:00:29Z) - Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions.
We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples.
We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
arXiv Detail & Related papers (2020-06-13T08:24:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.