Guess First to Enable Better Compression and Adversarial Robustness

• URL: http://arxiv.org/abs/2001.03311v1
• Date: Fri, 10 Jan 2020 05:12:22 GMT
• Title: Guess First to Enable Better Compression and Adversarial Robustness
• Authors: Sicheng Zhu, Bang An, Shiyu Niu
• Abstract summary: We propose a bio-inspired classification framework in which model inference is conditioned on label hypothesis. We provide a class of training objectives for this framework and an information bottleneck regularizer. Better compression and elimination of label information further bring better adversarial robustness without loss of natural accuracy.
• Score: 5.579029325265822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning models are generally vulnerable to adversarial examples, which is in contrast to the robustness of humans. In this paper, we try to leverage one of the mechanisms in human recognition and propose a bio-inspired classification framework in which model inference is conditioned on label hypothesis. We provide a class of training objectives for this framework and an information bottleneck regularizer which utilizes the advantage that label information can be discarded during inference. This framework enables better compression of the mutual information between inputs and latent representations without loss of learning capacity, at the cost of tractable inference complexity. Better compression and elimination of label information further bring better adversarial robustness without loss of natural accuracy, which is demonstrated in the experiment.

Related papers

• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• Prototype-Anchored Learning for Learning with Imperfect Annotations [83.7763875464011]
  It is challenging to learn unbiased classification models from imperfectly annotated datasets. We propose a prototype-anchored learning (PAL) method, which can be easily incorporated into various learning-based classification schemes. We verify the effectiveness of PAL on class-imbalanced learning and noise-tolerant learning by extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-06-23T10:25:37Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• Adversarial Robustness of Supervised Sparse Coding [34.94566482399662]
  We consider a model that involves learning a representation while at the same time giving a precise generalization bound and a robustness certificate. We focus on the hypothesis class obtained by combining a sparsity-promoting encoder coupled with a linear encoder. We provide a robustness certificate for end-to-end classification.
  arXiv  Detail & Related papers  (2020-10-22T22:05:21Z)
• Stylized Adversarial Defense [105.88250594033053]
  adversarial training creates perturbation patterns and includes them in the training set to robustify the model. We propose to exploit additional information from the feature space to craft stronger adversaries. Our adversarial training approach demonstrates strong robustness compared to state-of-the-art defenses.
  arXiv  Detail & Related papers  (2020-07-29T08:38:10Z)
• Derivation of Information-Theoretically Optimal Adversarial Attacks with Applications to Robust Machine Learning [11.206758778146288]
  We consider the theoretical problem of designing an optimal adversarial attack on a decision system. We present derivations of the optimal adversarial attacks for discrete and continuous signals of interest. We show that it is much harder to achieve adversarial attacks for minimizing mutual information when multiple redundant copies of the input signal are available.
  arXiv  Detail & Related papers  (2020-07-28T07:45:25Z)
• Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
  Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions. We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
  arXiv  Detail & Related papers  (2020-06-13T08:24:33Z)
• Self-Supervised Relational Reasoning for Representation Learning [5.076419064097733]
  In self-supervised learning, a system is tasked with achieving a surrogate objective by defining alternative targets on unlabeled data. We propose a novel self-supervised formulation of relational reasoning that allows a learner to bootstrap a signal from information implicit in unlabeled data. We evaluate the proposed method following a rigorous experimental procedure, using standard datasets, protocols, and backbones.
  arXiv  Detail & Related papers  (2020-06-10T14:24:25Z)
• Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization [15.087280646796527]
  Training machine learning models that are robust against adversarial inputs poses seemingly insurmountable challenges. We develop a notion of representation vulnerability that captures the maximum change of mutual information between the input and output distributions. We propose an unsupervised learning method for obtaining intrinsically robust representations by maximizing the worst-case mutual information.
  arXiv  Detail & Related papers  (2020-02-26T21:20:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.