A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification

• URL: http://arxiv.org/abs/2310.13990v1
• Date: Sat, 21 Oct 2023 12:35:48 GMT
• Title: A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
• Authors: Pierre Colombo, Nathan Noiry, Guillaume Staerman, Pablo Piantanida
• Abstract summary: One of the main application for such disentangled representations is fair classification. We adopt an information-theoretic view of this problem which motivates a novel family of regularizers. The resulting set of losses, called CLINIC, is parameter free and thus, it is easier and faster to train.
• Score: 46.884905701771004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One of the pursued objectives of deep learning is to provide tools that learn abstract representations of reality from the observation of multiple contextual situations. More precisely, one wishes to extract disentangled representations which are (i) low dimensional and (ii) whose components are independent and correspond to concepts capturing the essence of the objects under consideration (Locatello et al., 2019b). One step towards this ambitious project consists in learning disentangled representations with respect to a predefined (sensitive) attribute, e.g., the gender or age of the writer. Perhaps one of the main application for such disentangled representations is fair classification. Existing methods extract the last layer of a neural network trained with a loss that is composed of a cross-entropy objective and a disentanglement regularizer. In this work, we adopt an information-theoretic view of this problem which motivates a novel family of regularizers that minimizes the mutual information between the latent representation and the sensitive attribute conditional to the target. The resulting set of losses, called CLINIC, is parameter free and thus, it is easier and faster to train. CLINIC losses are studied through extensive numerical experiments by training over 2k neural networks. We demonstrate that our methods offer a better disentanglement/accuracy trade-off than previous techniques, and generalize better than training with cross-entropy loss solely provided that the disentanglement task is not too constraining.

Related papers

• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• Semantic Loss Functions for Neuro-Symbolic Structured Prediction [74.18322585177832]
  We discuss the semantic loss, which injects knowledge about such structure, defined symbolically, into training. It is agnostic to the arrangement of the symbols, and depends only on the semantics expressed thereby. It can be combined with both discriminative and generative neural models.
  arXiv  Detail & Related papers  (2024-05-12T22:18:25Z)
• DISCOVER: Making Vision Networks Interpretable via Competition and Dissection [11.028520416752325]
  This work contributes to post-hoc interpretability, and specifically Network Dissection. Our goal is to present a framework that makes it easier to discover the individual functionality of each neuron in a network trained on a vision task.
  arXiv  Detail & Related papers  (2023-10-07T21:57:23Z)
• Class Anchor Margin Loss for Content-Based Image Retrieval [97.81742911657497]
  We propose a novel repeller-attractor loss that falls in the metric learning paradigm, yet directly optimize for the L2 metric without the need of generating pairs. We evaluate the proposed objective in the context of few-shot and full-set training on the CBIR task, by using both convolutional and transformer architectures.
  arXiv  Detail & Related papers  (2023-06-01T12:53:10Z)
• Contrastive Classification and Representation Learning with Probabilistic Interpretation [5.979778557940212]
  Cross entropy loss has served as the main objective function for classification-based tasks. We propose a new version of the supervised contrastive training that learns jointly the parameters of the classifier and the backbone of the network.
  arXiv  Detail & Related papers  (2022-11-07T15:57:24Z)
• Efficient and Robust Classification for Sparse Attacks [34.48667992227529]
  We consider perturbations bounded by the $ell$--norm, which have been shown as effective attacks in the domains of image-recognition, natural language processing, and malware-detection. We propose a novel defense method that consists of "truncation" and "adrial training" Motivated by the insights we obtain, we extend these components to neural network classifiers.
  arXiv  Detail & Related papers  (2022-01-23T21:18:17Z)
• Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification [53.735029033681435]
  Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
  arXiv  Detail & Related papers  (2020-07-11T22:48:42Z)
• Feature Purification: How Adversarial Training Performs Robust Deep Learning [66.05472746340142]
  We show a principle that we call Feature Purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly gradient descent indeed this principle.
  arXiv  Detail & Related papers  (2020-05-20T16:56:08Z)
• T-Norms Driven Loss Functions for Machine Learning [19.569025323453257]
  A class of neural-symbolic approaches is based on First-Order Logic to represent prior knowledge. This paper shows that the loss function expressing these neural-symbolic learning tasks can be unambiguously determined.
  arXiv  Detail & Related papers  (2019-07-26T10:22:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.