Scalable Infomin Learning

• URL: http://arxiv.org/abs/2302.10701v1
• Date: Tue, 21 Feb 2023 14:40:25 GMT
• Title: Scalable Infomin Learning
• Authors: Yanzhi Chen, Weihao Sun, Yingzhen Li, Adrian Weller
• Abstract summary: infomin learning aims to learn a representation with high utility while being uninformative about a specified target. Recent works on infomin learning mainly use adversarial training, which involves training a neural network to estimate mutual information. We propose a new infomin learning approach, which uses a novel proxy metric to mutual information.
• Score: 39.77171117174905
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The task of infomin learning aims to learn a representation with high utility while being uninformative about a specified target, with the latter achieved by minimising the mutual information between the representation and the target. It has broad applications, ranging from training fair prediction models against protected attributes, to unsupervised learning with disentangled representations. Recent works on infomin learning mainly use adversarial training, which involves training a neural network to estimate mutual information or its proxy and thus is slow and difficult to optimise. Drawing on recent advances in slicing techniques, we propose a new infomin learning approach, which uses a novel proxy metric to mutual information. We further derive an accurate and analytically computable approximation to this proxy metric, thereby removing the need of constructing neural network-based mutual information estimators. Experiments on algorithmic fairness, disentangled representation learning and domain adaptation verify that our method can effectively remove unwanted information with limited time budget.

Related papers

• Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models [79.28821338925947]
  Domain-Class Incremental Learning is a realistic but challenging continual learning scenario. To handle these diverse tasks, pre-trained Vision-Language Models (VLMs) are introduced for their strong generalizability. This incurs a new problem: the knowledge encoded in the pre-trained VLMs may be disturbed when adapting to new tasks, compromising their inherent zero-shot ability. Existing methods tackle it by tuning VLMs with knowledge distillation on extra datasets, which demands heavy overhead. We propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of
  arXiv  Detail & Related papers  (2024-07-07T12:19:37Z)
• Discriminative Adversarial Unlearning [40.30974185546541]
  We introduce a novel machine unlearning framework founded upon the established principles of the min-max optimization paradigm. We capitalize on the capabilities of strong Membership Inference Attacks (MIA) to facilitate the unlearning of specific samples from a trained model. Our proposed algorithm closely approximates the ideal benchmark of retraining from scratch for both random sample forgetting and class-wise forgetting schemes.
  arXiv  Detail & Related papers  (2024-02-10T03:04:57Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Fortuitous Forgetting in Connectionist Networks [20.206607130719696]
  We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations.
  arXiv  Detail & Related papers  (2022-02-01T00:15:58Z)
• Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning [16.765461276790944]
  Self-adaptive training is a unified training algorithm that dynamically calibrates and enhances training process by model predictions without incurring extra computational cost. We analyze the training dynamics of deep networks on training data corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of emphany label information.
  arXiv  Detail & Related papers  (2021-01-21T17:17:30Z)
• Information Theoretic Meta Learning with Gaussian Processes [74.54485310507336]
  We formulate meta learning using information theoretic concepts; namely, mutual information and the information bottleneck. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning.
  arXiv  Detail & Related papers  (2020-09-07T16:47:30Z)
• Stylized Adversarial Defense [105.88250594033053]
  adversarial training creates perturbation patterns and includes them in the training set to robustify the model. We propose to exploit additional information from the feature space to craft stronger adversaries. Our adversarial training approach demonstrates strong robustness compared to state-of-the-art defenses.
  arXiv  Detail & Related papers  (2020-07-29T08:38:10Z)
• Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification [53.735029033681435]
  Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
  arXiv  Detail & Related papers  (2020-07-11T22:48:42Z)
• Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks [27.44348371795822]
  We develop a statistical minimax framework to characterize the limits of transfer learning. We derive a lower-bound for the target generalization error achievable by any algorithm as a function of the number of labeled source and target data.
  arXiv  Detail & Related papers  (2020-06-16T22:49:26Z)
• Self-Supervised Relational Reasoning for Representation Learning [5.076419064097733]
  In self-supervised learning, a system is tasked with achieving a surrogate objective by defining alternative targets on unlabeled data. We propose a novel self-supervised formulation of relational reasoning that allows a learner to bootstrap a signal from information implicit in unlabeled data. We evaluate the proposed method following a rigorous experimental procedure, using standard datasets, protocols, and backbones.
  arXiv  Detail & Related papers  (2020-06-10T14:24:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.