A Novel Estimator of Mutual Information for Learning to Disentangle
Textual Representations
- URL: http://arxiv.org/abs/2105.02685v1
- Date: Thu, 6 May 2021 14:05:06 GMT
- Title: A Novel Estimator of Mutual Information for Learning to Disentangle
Textual Representations
- Authors: Pierre Colombo and Chloe Clavel and Pablo Piantanida
- Abstract summary: This paper introduces a novel variational upper bound to the mutual information between an attribute and the latent code of an encoder.
It aims at controlling the approximation error via the Renyi's divergence, leading to both better disentangled representations and a precise control of the desirable degree of disentanglement.
We show the superiority of this method on fair classification and on textual style transfer tasks.
- Score: 27.129551973093008
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Learning disentangled representations of textual data is essential for many
natural language tasks such as fair classification, style transfer and sentence
generation, among others. The existent dominant approaches in the context of
text data {either rely} on training an adversary (discriminator) that aims at
making attribute values difficult to be inferred from the latent code {or rely
on minimising variational bounds of the mutual information between latent code
and the value attribute}. {However, the available methods suffer of the
impossibility to provide a fine-grained control of the degree (or force) of
disentanglement.} {In contrast to} {adversarial methods}, which are remarkably
simple, although the adversary seems to be performing perfectly well during the
training phase, after it is completed a fair amount of information about the
undesired attribute still remains. This paper introduces a novel variational
upper bound to the mutual information between an attribute and the latent code
of an encoder. Our bound aims at controlling the approximation error via the
Renyi's divergence, leading to both better disentangled representations and in
particular, a precise control of the desirable degree of disentanglement {than
state-of-the-art methods proposed for textual data}. Furthermore, it does not
suffer from the degeneracy of other losses in multi-class scenarios. We show
the superiority of this method on fair classification and on textual style
transfer tasks. Additionally, we provide new insights illustrating various
trade-offs in style transfer when attempting to learn disentangled
representations and quality of the generated sentence.
Related papers
- Fair Text Classification with Wasserstein Independence [4.887319013701134]
Group fairness is a central research topic in text classification, where reaching fair treatment between sensitive groups remains an open challenge.
This paper presents a novel method for mitigating biases in neural text classification, agnostic to the model architecture.
arXiv Detail & Related papers (2023-11-21T15:51:06Z) - Learning Fair Representation via Distributional Contrastive
Disentanglement [9.577369164287813]
Learning fair representation is crucial for achieving fairness or debiasing sensitive information.
We propose a new approach, learning FAir Representation via distributional CONtrastive Variational AutoEncoder (FarconVAE)
We show superior performance on fairness, pretrained model debiasing, and domain generalization tasks from various modalities.
arXiv Detail & Related papers (2022-06-17T12:58:58Z) - Annotation Error Detection: Analyzing the Past and Present for a More
Coherent Future [63.99570204416711]
We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets.
We define a uniform evaluation setup including a new formalization of the annotation error detection task.
We release our datasets and implementations in an easy-to-use and open source software package.
arXiv Detail & Related papers (2022-06-05T22:31:45Z) - Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning.
Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
arXiv Detail & Related papers (2022-05-23T17:38:30Z) - Learning Disentangled Textual Representations via Statistical Measures
of Similarity [35.74568888409149]
We introduce a family of regularizers for learning disentangled representations that do not require training.
Our novel regularizers do not require additional training, are faster and do not involve additional tuning.
arXiv Detail & Related papers (2022-05-07T08:06:22Z) - Conditional Contrastive Learning: Removing Undesirable Information in
Self-Supervised Representations [108.29288034509305]
We develop conditional contrastive learning to remove undesirable information in self-supervised representations.
We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks.
arXiv Detail & Related papers (2021-06-05T10:51:26Z) - Scaling-up Disentanglement for Image Translation [40.7636450847048]
We propose OverLORD, a single framework for disentangling labeled and unlabeled attributes.
We do not rely on adversarial training or any architectural biases.
In an extensive evaluation, we present significantly better disentanglement with higher translation quality and greater output diversity than state-of-the-art methods.
arXiv Detail & Related papers (2021-03-25T17:52:38Z) - Improving Disentangled Text Representation Learning with
Information-Theoretic Guidance [99.68851329919858]
discrete nature of natural language makes disentangling of textual representations more challenging.
Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text.
Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation.
arXiv Detail & Related papers (2020-06-01T03:36:01Z) - Contextual Text Style Transfer [73.66285813595616]
Contextual Text Style Transfer aims to translate a sentence into a desired style with its surrounding context taken into account.
We propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context.
Two new benchmarks, Enron-Context and Reddit-Context, are introduced for formality and offensiveness style transfer.
arXiv Detail & Related papers (2020-04-30T23:01:12Z) - Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
We propose a novel disentanglement approach to invariant representation problem.
We enforce the meaningful representation to be agnostic to sensitive information by entropy.
The proposed approach is evaluated on five publicly available datasets.
arXiv Detail & Related papers (2020-03-12T11:09:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.